Obtaining reddit data

I am interested in obtaining data from different reddit subreddits. Does anyone know

if there is a reddit/other api similar like twitter does to crawl all the pages?


Yes, reddit has an API that can be used for a variety of purposes such as data

collection, automatic commenting bots, or even to assist in subreddit moderation.

There are a few places to discover information on reddit's API:

    github reddit wiki -- provides the overview and rules for using reddit's API

(follow the rules)
    automatically generated API docs -- provides information on the requests needed to

access most of the API endpoints
    /r/redditdev -- the reddit community dedicated to answering questions both about

reddit's source code and about reddit's API

If there is a particular programming language you are already familiar with, you

should check out the existing set of API wrappers for various languages. Despite my

bias (I am the package maintainer) I am quite certain PRAW, for python, has support

for the largest number of reddit API features.



Source: http://stackoverflow.com/questions/14322834/obtaining-reddit-data

Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Data Recovery Services - Be Wary of Cheap Prices

Data recovery is a specialized, complicated process. Proper hard drive recovery can require manipulation of data at the sector level, transplantation of internal components and various other procedures. These techniques are very involved and require not only talented, knowledgeable technicians, but also an extensive inventory of disk drives to use for parts when necessary and clean facilities to conduct the work.

Unfortunately these factors mean that, in most cases, recovery services are quite expensive. Technician training, hard drive inventories and special equipment all come with a cost.

If you search for disk recovery services, you will likely find several smaller companies that offer hard disk data recovery for a fraction of the prices usually quoted by larger, more experienced organizations. These companies often operate from small offices or, in some cases, private homes. They do not possess clean room facilities, large disk drive inventories or many other pieces of equipment necessary to successfully complete most hard drive recovery cases.

When you take into account all of the training, parts and facilities necessary, you can see how it is impossible for a company to charge $200 for a hard drive recovery and not operate at a loss.

What these companies usually do is run a software program on the hard disk drive. Sometimes, if there is no physical damage to the disk drive, this program is able to recover some of the data. However, hard disk data recovery is much more than just running software. No software can recover data from a hard drive that has failed heads, damaged platters or electrical damage. In fact, attempting to run a hard drive that is physically damaged can make the problem worse. Trying to use software to recover data from a hard drive with failed read/write heads, for example, can lead to the heads scraping the platters of the drive and leaving the data unrecoverable.

Another way these smaller companies conduct business is by forwarding data recovery cases they cannot recover to larger organizations. Of course, the smaller companies charge their clients a premium for this service. In these cases it would have actually been cheaper to use the larger company in the first place.

You will also likely find that many smaller recovery companies charge evaluation or diagnostic fees upfront. They charge these fees regardless of whether or not any data is recovered. In many cases clients desperate to recover their data pay these fees and get nothing in return but a failed recovery. Smaller data recovery services simply do not have the skills, training, facilities and parts to successfully recover most disk drives. It is more cost efficient for them to make one attempt at running a software program and then call the case unsuccessful.

Sometimes you may get lucky working with a smaller data recovery company, but in most cases you will end up paying for a failed recovery. In the worst case scenario you could end up with a damaged hard drive that is now unrecoverable by any data recovery service.

You will waste time and money working with these services. You could even lose your valuable data for good.

If your data is important enough to consider data recovery, it is important enough to seek a reputable, skilled data recovery company. All major data recovery services offer free evaluations and most do not charge clients for unsuccessful recoveries. Sometimes you only have one shot to recover data on a disk drive before the platters are seriously damaged and the data is lost for good. Taking chances with inexperienced companies is not worth the risk.

Source:http://ezinearticles.com/?Data-Recovery-Services---Be-Wary-of-Cheap-Prices&id=4706055

Collecting and Managing Data for Small Businesses

Is your company doing well? Are you profitable? How much longer will it be until you are? Where can you cut expenses? What are your most profitable products or services? Which are the least profitable? What the shelf-life of a particular product? How effective is your marketing strategy?

As business leaders, this type of information is the type of information that we need to have on hand in order to make decisions about the company. Planning solely on what is in the bank, or focusing only on one aspect of what makes your company a) successful or b) keeps it out of closing is a very poor way of operating. This is pure tunnel vision.

If you consider, for a moment, two tools that are widely used in the business world - Porter's Five Forces and the SWOT analysis, you'll notice that part of the analysis is based on things that impact the business - are outside of the business's control. As business leaders, you know that strategically, this cannot occur just in exercise, but must exist in the way that you do business. Monitoring, making adjustments and acting must be an ongoing mentality if your goal is to build an extraordinary business. Several recommendations we have made to clients include:

    Understand what questions you want to answer. Here are some samples:

    How do we know when we can purchase a new building or expand capacity

    How do we know how effective our sales people are

    How do we know how effective our marketing and other business development activities are

    Understand what kind of data you need to collect in order to make your decisions. Typically, these are going to be things such as your financials - sales, cost of goods sold, expenses, profit, investments, interest and taxes, your business development activities, manufacturing costs and rates, etc.

    Determine how to collect the data - including what is feasible. Take into consideration how you operate - does it need to be mobile? does the information need to be housed in a cloud?

    Determine how the data needs to be delivered. If you have a ton of data and like to drill down from "high level" analysis down to the details, perhaps you want something more visual. If you like to play with the numbers yourself and run scenarios, perhaps you like to play with the raw data.

•    Decide how much your level of investment. Consider this:

•    No Investment - If you don't get the data to make decisions, the likelihood of success is minimalized.

•    Your Time - If you collect and mine the data yourself, what else could or should you be doing to build or grow the business.

•    Your Resources - You could have a qualified employee collect and mine the data for you.

•    Your Money - You could invest in a software solution - be it customized, off the shelf, or a combination of the two - that could collect the data. We have used and recommend a product called Work, Etc., to centrally house most transaction that occur in the business in order to give us a single data-collection source.

    Research solutions that will work best for your company, considering the factors we pointed out above. You may consider other factors that are specific to your company, such as the ability to sync with certain existing solutions or the ability to be housed on a centrally located or remote server, etc.. May companies look at Open Source solutions such as xTurple or SugarCRM. Despite your choice, consider the total cost of ownership of the solution before investing in it.

    Determine how the solution will be implemented and how training on the software will roll out. This may mean hiring a consultant, going to a class or classes, investing in an online training solution or spending time with tech support for a self-install. Consider the different types of investment.

    Plan to re-enforce the need to use whatever solution is recommended. Change takes time. Habits take time. By providing some structure, you will increase the likelihood of success.

Your company's ability to collect and decipher the data from the activities in and around your company can be the determine factor between a series of successful ventures and a series of hit-or-miss activities. Even the simplest data-collection activities should help you determine your company's path.

Source:http://ezinearticles.com/?Collecting-and-Managing-Data-for-Small-Businesses&id=6923386

Data Entry Outsourcing - Important Part of Data Entry Services

It is rightly said if you try to manage everything you end up managing nothing. The business environment, these days, is very competitive. Requirements and technology change by a wink of an eye. It is imperative for any business to keep abreast of the technological and business environment changes. This requires a more focused and strategic approach towards primary function of the business.

Over the years the corporate have become selective on the business processes they put their time and money on. This calls for an arrangement where in the managers can concentrate on their core competencies and get the supporting operations done by someone else. The answer to all this is Outsourcing. Outsourcing means getting some portion of the service, operations, or business done through some other organization that is more familiar with this work.

Data is by far the most important asset of any business. However, before businesses could use the data, it has to be fed in to the computers. Mostly businesses outsource this act of Data Processing for multiple benefits. Firstly, the companies could do away with this tedious task and hence could invest their time and effort on more important things. Secondly, the company or the freelancer to whom the work is outsourced is a specialist in the job and hence the work is completed with more precision and with lower turnaround time. Another benefit is that the data entry work might be for some duration and need not continue for a life time, in such a scenario the company gets away from arranging people and infrastructure for the data entry job.

Having discussed the need to outsource the Data, another important aspect to cover in this domain is the efficiency of the agency to which the work is outsourced. Business houses should do their homework before selecting the service provider. Apart from the work experience and the quality of the people, confidentiality should always be kept on top of the agenda. Business houses may have at face the brunt In case of leakage or misuse of the sensitive information. It is imperative for the service providers to ensure the best security measures at the work place.

Source:http://ezinearticles.com/?Data-Entry-Outsourcing---Important-Part-of-Data-Entry-Services&id=5255220

Law of Cards: Beckett Sues Two More for Scraping Data

The recent trend in trading card cases focuses more on protecting trading card data than on trading cards themselves. Beckett has filed two more lawsuits over alleged data scraping.

Collectors Universe (CU) started this trend with lawsuits against MWP Software/Matthew Perry and SaintSoft LLC/John Kountz. In both of these suits, CU alleged the defendants copied CU data available only to paying CU subscribers, and then unlawfully (and in violation of CU's terms of service) used that data on the defendants' websites or in their own apps.

Pokémon then followed by suing the Pokellector (a website which attempts to provide a comprehensive list of all Pokémon cards, along with images of those cards) for allegedly reproducing copyrighted images of its cards without permission, trademark infringement and unlawfully rebranding the images of Pokémon cards with a Pokellector trademark.

Beckett continued the trend by bringing a trade secret lawsuit against Check Out My Cards (COMC) which, among other things, made a big deal out of COMC's "scraping" of checklists and pricing data (although COMC alleges such scraping may have been authorized).

On April 24, Beckett filed two more lawsuits focusing on pricing and checklist data. This time against Custom Plush Productions Inc's The Card Collector and Derek Miller of Miller Cards.

These two new complaints cannibalize a lot of the COMC complaint and are nearly mirror images of each other as well.

It also looks like Beckett might have let an intern from the marketing department edit these complaints. Paragraph six of both state, "Beckett is the preeminent company and most trusted source operating within the collectibles industry, and it has been since it formed in 1984."

Sounds more like a PR piece than a lawsuit, right?

For The Card Collector case, Beckett alleges The Card Collector accessed Beckett's pricing guides through the Beckett website and unlawfully "scraped" Beckett's copyrightable checklist and pricing data. Allegedly, The Card Collector now offers Beckett's card checklists, grading and pricing data on its own website and through a CD-ROM it sells to others. For that Beckett brings claims for copyright infringement, common law misappropriation, "accounting" (bring out the accountants!), unjust enrichment, a violation of Computer Fraud and Abuse Act, trespass and a Harmful Access by Computer Under Texas Penal Code.

A little bit of everything, right?

For the second new suit, Beckett alleges Miller Cards operates several websites (such as millercards.net and freebaseballcardspriceguide.com) that provide checklists and pricing guides for card collectors. Beckett contends that Miller accessed Beckett's price guides through his Beckett membership, violated the membership agreement by "scraping" this data and then unlawfully reproduced the data on his websites. It brings the identical claims against Miller Cards that it brought against the Card Collector.

Giving all of this scraping going on, I guess, the next planking or Tebowing-type meme should be the "scraping" Beckett meme. I'm looking forward to seeing your pictures on what that should look like.

Now, we've only seen Beckett's side of the story, but it looks bad for both defendants. Sure, there are issues about how much copyright protection should be afforded checklists, but these cases are not just about publicly available checklist data. The checklist and pricing information from these cases was allegedly scraped from behind a pay wall at and then distributed to others in violation of Beckett's terms of service.

Allegedly stealing something from behind a pay wall to give or sell to others just doesn’t look good.

Given this, in all likelihood, these cases will disappear (settle) quickly before we get into any real issues.

These two cases (and the COMC case) have another implication for the industry. Beckett has a template complaint that it looks like it's going to recycle again and again against scrapers. And it's policing the Internet looking for infringers. I predict then we'll see this complaint filed again this year, perhaps multiple times.

If you'd like to review a copy of these complaints, click here for the Card Collector complaint and here for the Miller Cards complaint.

The information provided in Paul Lesko's "Law of Cards" column is not intended to be legal advice, but merely conveys general information related to legal issues commonly encountered in the sports industry. This information is not intended to create any legal relationship between Paul Lesko, the Simmons Browder Gianaris Angelides & Barnerd LLC or any attorney and the user. Neither the transmission nor receipt of these website materials will create an attorney-client relationship between the author and the readers.

The views expressed in the "Law of Cards" column are solely those of the author and are not affiliated with the Simmons Law Firm. You should not act or rely on any information in the "Law of Cards" column without seeking the advice of an attorney. The determination of whether you need legal services and your choice of a lawyer are very important matters that should not be based on websites or advertisements.

Source: http://www.cardboardconnection.com/news/law-of-cards-beckett-sues-two-more-for-scraping-data