PDF scraping using R

I have been using the XML package successfully for extracting HTML tables but want to extend to PDF's. From previous questions it does not appear that there is a simple R solution but wondered if there had been any recent developments

Failing that, is there some way in Python (in which I am a complete Novice) to obtain and manipulate pdfs so that I could finish the job off with the R XML package

Extracting text from PDFs is hard, and nearly always requires lots of care.

I'd start with the command line tools such as pdftotext and see what they spit out. The problem is that PDFs can store the text in any order, can use awkward font encodings, and can do things like use ligature characters (the joined up 'ff' and 'ij' that you see in proper typesetting) to throw you.

pdftotext is installable on any Linux system



Source: http://stackoverflow.com/questions/7918718/pdf-scraping-using-r

Obtaining reddit data

I am interested in obtaining data from different reddit subreddits. Does anyone know

if there is a reddit/other api similar like twitter does to crawl all the pages?


Yes, reddit has an API that can be used for a variety of purposes such as data

collection, automatic commenting bots, or even to assist in subreddit moderation.

There are a few places to discover information on reddit's API:

    github reddit wiki -- provides the overview and rules for using reddit's API

(follow the rules)
    automatically generated API docs -- provides information on the requests needed to

access most of the API endpoints
    /r/redditdev -- the reddit community dedicated to answering questions both about

reddit's source code and about reddit's API

If there is a particular programming language you are already familiar with, you

should check out the existing set of API wrappers for various languages. Despite my

bias (I am the package maintainer) I am quite certain PRAW, for python, has support

for the largest number of reddit API features.



Source: http://stackoverflow.com/questions/14322834/obtaining-reddit-data

Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Data Recovery Services - Be Wary of Cheap Prices

Data recovery is a specialized, complicated process. Proper hard drive recovery can require manipulation of data at the sector level, transplantation of internal components and various other procedures. These techniques are very involved and require not only talented, knowledgeable technicians, but also an extensive inventory of disk drives to use for parts when necessary and clean facilities to conduct the work.

Unfortunately these factors mean that, in most cases, recovery services are quite expensive. Technician training, hard drive inventories and special equipment all come with a cost.

If you search for disk recovery services, you will likely find several smaller companies that offer hard disk data recovery for a fraction of the prices usually quoted by larger, more experienced organizations. These companies often operate from small offices or, in some cases, private homes. They do not possess clean room facilities, large disk drive inventories or many other pieces of equipment necessary to successfully complete most hard drive recovery cases.

When you take into account all of the training, parts and facilities necessary, you can see how it is impossible for a company to charge $200 for a hard drive recovery and not operate at a loss.

What these companies usually do is run a software program on the hard disk drive. Sometimes, if there is no physical damage to the disk drive, this program is able to recover some of the data. However, hard disk data recovery is much more than just running software. No software can recover data from a hard drive that has failed heads, damaged platters or electrical damage. In fact, attempting to run a hard drive that is physically damaged can make the problem worse. Trying to use software to recover data from a hard drive with failed read/write heads, for example, can lead to the heads scraping the platters of the drive and leaving the data unrecoverable.

Another way these smaller companies conduct business is by forwarding data recovery cases they cannot recover to larger organizations. Of course, the smaller companies charge their clients a premium for this service. In these cases it would have actually been cheaper to use the larger company in the first place.

You will also likely find that many smaller recovery companies charge evaluation or diagnostic fees upfront. They charge these fees regardless of whether or not any data is recovered. In many cases clients desperate to recover their data pay these fees and get nothing in return but a failed recovery. Smaller data recovery services simply do not have the skills, training, facilities and parts to successfully recover most disk drives. It is more cost efficient for them to make one attempt at running a software program and then call the case unsuccessful.

Sometimes you may get lucky working with a smaller data recovery company, but in most cases you will end up paying for a failed recovery. In the worst case scenario you could end up with a damaged hard drive that is now unrecoverable by any data recovery service.

You will waste time and money working with these services. You could even lose your valuable data for good.

If your data is important enough to consider data recovery, it is important enough to seek a reputable, skilled data recovery company. All major data recovery services offer free evaluations and most do not charge clients for unsuccessful recoveries. Sometimes you only have one shot to recover data on a disk drive before the platters are seriously damaged and the data is lost for good. Taking chances with inexperienced companies is not worth the risk.

Source:http://ezinearticles.com/?Data-Recovery-Services---Be-Wary-of-Cheap-Prices&id=4706055

Collecting and Managing Data for Small Businesses

Is your company doing well? Are you profitable? How much longer will it be until you are? Where can you cut expenses? What are your most profitable products or services? Which are the least profitable? What the shelf-life of a particular product? How effective is your marketing strategy?

As business leaders, this type of information is the type of information that we need to have on hand in order to make decisions about the company. Planning solely on what is in the bank, or focusing only on one aspect of what makes your company a) successful or b) keeps it out of closing is a very poor way of operating. This is pure tunnel vision.

If you consider, for a moment, two tools that are widely used in the business world - Porter's Five Forces and the SWOT analysis, you'll notice that part of the analysis is based on things that impact the business - are outside of the business's control. As business leaders, you know that strategically, this cannot occur just in exercise, but must exist in the way that you do business. Monitoring, making adjustments and acting must be an ongoing mentality if your goal is to build an extraordinary business. Several recommendations we have made to clients include:

    Understand what questions you want to answer. Here are some samples:

    How do we know when we can purchase a new building or expand capacity

    How do we know how effective our sales people are

    How do we know how effective our marketing and other business development activities are

    Understand what kind of data you need to collect in order to make your decisions. Typically, these are going to be things such as your financials - sales, cost of goods sold, expenses, profit, investments, interest and taxes, your business development activities, manufacturing costs and rates, etc.

    Determine how to collect the data - including what is feasible. Take into consideration how you operate - does it need to be mobile? does the information need to be housed in a cloud?

    Determine how the data needs to be delivered. If you have a ton of data and like to drill down from "high level" analysis down to the details, perhaps you want something more visual. If you like to play with the numbers yourself and run scenarios, perhaps you like to play with the raw data.

•    Decide how much your level of investment. Consider this:

•    No Investment - If you don't get the data to make decisions, the likelihood of success is minimalized.

•    Your Time - If you collect and mine the data yourself, what else could or should you be doing to build or grow the business.

•    Your Resources - You could have a qualified employee collect and mine the data for you.

•    Your Money - You could invest in a software solution - be it customized, off the shelf, or a combination of the two - that could collect the data. We have used and recommend a product called Work, Etc., to centrally house most transaction that occur in the business in order to give us a single data-collection source.

    Research solutions that will work best for your company, considering the factors we pointed out above. You may consider other factors that are specific to your company, such as the ability to sync with certain existing solutions or the ability to be housed on a centrally located or remote server, etc.. May companies look at Open Source solutions such as xTurple or SugarCRM. Despite your choice, consider the total cost of ownership of the solution before investing in it.

    Determine how the solution will be implemented and how training on the software will roll out. This may mean hiring a consultant, going to a class or classes, investing in an online training solution or spending time with tech support for a self-install. Consider the different types of investment.

    Plan to re-enforce the need to use whatever solution is recommended. Change takes time. Habits take time. By providing some structure, you will increase the likelihood of success.

Your company's ability to collect and decipher the data from the activities in and around your company can be the determine factor between a series of successful ventures and a series of hit-or-miss activities. Even the simplest data-collection activities should help you determine your company's path.

Source:http://ezinearticles.com/?Collecting-and-Managing-Data-for-Small-Businesses&id=6923386

Data Entry Outsourcing - Important Part of Data Entry Services

It is rightly said if you try to manage everything you end up managing nothing. The business environment, these days, is very competitive. Requirements and technology change by a wink of an eye. It is imperative for any business to keep abreast of the technological and business environment changes. This requires a more focused and strategic approach towards primary function of the business.

Over the years the corporate have become selective on the business processes they put their time and money on. This calls for an arrangement where in the managers can concentrate on their core competencies and get the supporting operations done by someone else. The answer to all this is Outsourcing. Outsourcing means getting some portion of the service, operations, or business done through some other organization that is more familiar with this work.

Data is by far the most important asset of any business. However, before businesses could use the data, it has to be fed in to the computers. Mostly businesses outsource this act of Data Processing for multiple benefits. Firstly, the companies could do away with this tedious task and hence could invest their time and effort on more important things. Secondly, the company or the freelancer to whom the work is outsourced is a specialist in the job and hence the work is completed with more precision and with lower turnaround time. Another benefit is that the data entry work might be for some duration and need not continue for a life time, in such a scenario the company gets away from arranging people and infrastructure for the data entry job.

Having discussed the need to outsource the Data, another important aspect to cover in this domain is the efficiency of the agency to which the work is outsourced. Business houses should do their homework before selecting the service provider. Apart from the work experience and the quality of the people, confidentiality should always be kept on top of the agenda. Business houses may have at face the brunt In case of leakage or misuse of the sensitive information. It is imperative for the service providers to ensure the best security measures at the work place.

Source:http://ezinearticles.com/?Data-Entry-Outsourcing---Important-Part-of-Data-Entry-Services&id=5255220