To do so with the greatest value, you need not only document conversion using Optical Character Recognition ( OCR ), but some other advanced imaging and recognition tools. In the digital document world, you don’t have only the data contained in the document, you have various other meta data items such as file name, file location ( taxonomy ), tags, etc. In order to marry paper with digital the same has to be duplicated on the paper document and has to occur at time of document processing. This could be a manual process or automated, and depending on your paper volume doing it in manual may be OK. To compete with the efficiency of digital documents however, automatic is the way to go.

Using OCR, image-based and contextual-based classification, paper or image documents that enter the workflow can obtain the same value as digital documents. The OCR is responsible for getting all the content from the document. The purpose of this content is for search, indexing, auto-filing, as well as generation of keywords ( tags ) associated with a taxonomy. In order to determine where the document fits into a taxonomy, you must first classify it.

For classification to be most effective, it happens on two levels. Image-based classification, which is what the document looks like, classifies documents based on their physical structure which is a good indicator of its type and very fast. Contextual classification, which is what words are contained in the document, is one level deeper in classification and looks for the keywords that would make a document one type over another.  For some environments, image-based classification can do the job entirely.  Once classification is known, a classification engine can place the document in the correct spot in an existing taxonomy. Once an ID or classification is determined, it is no challenge to apply tags, file-naming, and file location to a document.

Workflow can stand alone, but injected with the power of OCR and document classification, it becomes a power house that does not know the difference between paper and digital.

Chris Riley – About

The latest updates to SharePoint are looking more and more like a fully functional content management system, with one exception, they focus on the general use only. Microsoft’s methodology on the product is to create the functionality that everyone requires and leave the reset to the users or partners. One such feature that is not the general is imaging and OCR. While this would not be the first time that I saw Microsoft indicate they would not delve into an arena, only a few years later finding them as a threat in it. I believe that Microsoft at this point will not invest much development in the imaging, data capture, and OCR functionality to SharePoint.

While there are several imaging packages out there that do export directly to SharePoint, they all share the primary problem of supporting newer versions. When SharePoint 2010 comes out, it means substantial development for these vendors to improve their products. With the current economic situation I expect many to refuse.

The good news is the number of hot-folder driven conversion applications that exist is high, and how hot-folders integrate nicely with all applications makes them an ideal solution. Utilizing an existing imaging or OCR platform as a server based watch folder process allows companies to integrate OCR and data capture functionality into SharePoint in a day simply by having the documents converted prior to an upload.

SharePoint is not going away, and the need to get searchable images into the system is clear. Until the time Microsoft decides to invest money here, it’s time to find a stable, and scalable way to OCR documents prior to SharePoint import.

Chris Riley – About

First is initial image quality. Without yourself identifying any objects on the page, look objectively at the document as a collection of questionable objects and see if you think the image quality is good. This is determined by coherence of each object. Are object borders tight and determinable? Are there objects interfering with other objects? Is the background of the image significantly different than all objects?

Second am identification of objects. Find text, graphics, lines, paragraphs, etc. Are their borders far enough apart? Is their type clear? This is most important for text. Is their printing consistent? For example does text go from one background color to another, this would make it inconsistent. Or another example does the straightness of lines change throughout the document? And can one object be confused for another?

And third, now that you know the objects, how easy is it to determine their value. Is the value obvious? Do you have to look at it for a while to figure it out?

Essentially the three above steps are exactly what the conversion ( OCR, ICR, OMR ) product does in order to read a document. With field level recognition it’s a bit more elaborate, but the core is the same. By identifying early on what the anticipated accuracy is of a document, you can then adjust your scan, or input settings accordingly even before looking at any technology. Doing this will give the best chance for success.

Chris Riley – About

In-line OCR is used primarily for mail-room processing on high speed high volume scanners, or on manufacturing assembly lines. Both scenarios need data from the input asset quickly. The benefit’s of in-line OCR is it’s the fastest OCR around. Usually the OCR is apart of firmware, and optimized for speed. If you imagine an assembly line of bottles, the bottles pass the camera at millisecond time. To wait for OCR would be a huge bottleneck in the quality control and inventory process. The downside to in-line OCR is accuracy. Usually in the case of the assembly line the engine has been so tuned that it is extremely accurate for a single image type. Where accuracy is proven to be less, is when it comes to document scanning, the digital mail-room. In the digital mail-room the in-line OCR, in order to be as fast as it is, must be an engine that is reduced in complexity, namely removing document analysis and reading of complex fonts. Because of this, when documents are scanned the accuracy cannot compare to that of PC based OCR.

PC based OCR has the benefit of scalability. It can work on the widest range of document types. Also because it’s using the PC, it has the latest and greatest technologies that work on degraded documents and complex documents. The downside of PC based OCR is that it’s not as fast as in-line. 99.9% it is fast enough. Many times PC based OCR is used at document scanner rated speed of 60 pages a minute. This is plenty fast for those who’s primary concern is quality. It is not fast enough for machine to machine hand-off’s, but this is not its primary use.

You may never encounter in-line OCR, but knowing about the technology helps understand the world of recognition and applications of such technology.

Chris Riley – About

The type of compression that I’m discussing here is file specific compression.  These are compression technologies that operate on single file types, and have special algorithms to reduce the size of those file types.  The two most common examples are JPEG image files and PDF files.  Using type specific compression has the benefit of being able to manipulate the files as you would normally.  The opposite of type specific is compression technologies such as Zip or Tar.  Here you have to uncompress the files before utilizing them.

Because the file types are left intact with type specific compression, it means that you can email the files after compression, search engines can index them, and they can be opened in your typical viewer.  The reality is that hard drive space is cheap and adding more is relatively easy.  So for some, compression is more about efficiency.  With proper compression, emails are sent and received faster, search engines crawl faster and indexes are smaller, and opening large files takes less time.

This is not to diminish the use of compression to save space in an ever increasing data collection world.  The purpose of this article is to highlight the other and substantial benefits of type specific file compression.  The trick now becomes finding the right compression tools that create high quality compressed files and compatible with typical file browsers.

Chris Riley – About

Set it and forget it OCR is where you take your OCR product and configure it to automatically process any images that appear in a certain folder. For my office, I scan to an “input” folder and all the resulting compressed and OCR’ed PDF files end up in the “File Cabinet” folder. My strategy will not work for the timid because basically I’m relying solely on the power of OCR text and search to retrieve documents when I need them. Most would rather configure their ADF scanner to have a setting or folder for each particular class of documents. Most document scanners anymore have as few as 9 and as many as 99 destinations you can program. You can set each destination as its own input folder with its own OCR settings with its own output folder.

I know I can do this because I know what settings it takes to get the quality of OCR I would need to at least have one or more usable keyword on the document for search.  And after-all, I’m an expert in OCR so to not use it everyday would be crazy in its own right. I’ve yet to be proven wrong, my “File Cabinet” abyss has always given me the information I need at the time I asked for it and sometimes even new information I did not realize I had.

Now for you records management folks shaking your head, I understand your complaint. It should not be about my approach but should be about what I do with the final paper product. For those items that are for legal or business reasons that are deemed as a record by your taxonomy, they should be filed as such, perhaps scanned again as a record, and for heavens sake if you are not supposed to, don’t destroy it!

The purpose of my madness is to touch paper as little as possible, and get information only when I need it. I am an extremist, but I assure you there is serious value, and a little fun in the set it and forget it OCR technique.

Chris Riley – About

As we create files on our computer, access them, move them, modify them, we are fragmenting the drive. Overly fragmented drives slow down machines and increase risk for damage and corruption. The more files you have, the more this multiplies. Real-time file compression helps with this because as soon as a file is generated, it’s compressed. There is less space being used, and the need to compress in the future is gone. Back-log compression ( compressing in bulk of all your files ) requires a lot of activity on the hard drive and increases the fragmentation. The other risk of bulk conversion is the fact that you only have one chance to get it right.

Bad compression is not just an irritation, it’s a risk. Usually when you compress a file, you are removing the original. The whole purpose is to save space, not use up more by keeping both copies. But because of the need to make sure you are compressing the file correctly, keeping both files waste a lot of space. When doing day-forward compression or real-time compression it’s easy to check as the files come across to make sure at initial setup everything is good, but if you do bulk compression and make a mistake you could have ruined a large library of files.

I firmly believe in file compression, but I know first hand the risk of doing it incorrectly. I now compress files as they are created and no longer have to think about data piling up faster then I can find ways to save space.

Chris Riley – About

You may have seen in a previous post where I eluded to the possibilities of using OCR to scrape screen-shots. This is one of the best real examples of why the technology is so useful. When you don’t have XML and ODBC or any of the other great standards that allow the exchange of data from one system to another, you always have what you can see, and if you can see it you can OCR it. If you can view the data on the screen, you can move it to a new system.

Using OCR to either problematically or manual read portions of a screen where the legacy system window is displaying data, copy it to memory, and paste it into the new system is one of the most ingenious ways to ensure the neutrality of your data. Vendor lock down attempts, or old technology should not prevent you from getting to what you own, the information.

Whether it’s a manual process or a programmatic one, the ability to OCR screen-shots and to migrate data is the hidden secret to crack any proprietary software safe.

Chris Riley – About

The reality is that paper consumption has not really decreased. In some areas paper has been replaced with electronic data interchange EDI, but in other areas it has actually increased. Studies have also shown that because documents are being scanned more often, there is also an increase in printing when the documents need to be shared or re-purposed. But I’m not here to argue that paper is not going away and that document conversion technologies are required to convert them. I’m here to point out a few futuristic uses of the technology that technologists like to already talk about and involve OCR.

Data Security

The first futuristic use of the technology that I would like to discuss is the use of OCR in data security. Text strings sent over the Internet are far easier to sniff and unlock than a compressed JPEG image. What if you were to convert the text into a JPEG during transmission and the person on the receiving end would OCR it to get the data. By doing so the data has been masked in a more efficient and secretive way. For added security, proprietary image formats could be devised.

File Compression

Storing ASCII text takes up far less space than an image or video file. As apart of the future of compression technologies, expect that OCR will be uesd to extract the text from an image and saved as an ASCII file. Viewers will convert the text back to an image during viewing. This then removes the image portion of the text and significantly reduces file size.

Robots

How else to you expect future robots to read text? OCR of course. The eyes of the robot are essentially a camera that takes pictures of images rapidly. When the robot is faced with the comprehension of text, the image will be converted using OCR and fed through an engine to gain meaning from the text and act on it.

So there you have it, three really cool and cutting edge ways OCR is and will be used in the future. Paper is not going away, but even if it were,  just look at the other cool uses of OCR technology.

Chris Riley – About

In my office, paper comes in, is reviewed for value, gets scanned, and shredded or filed. I have setup a system that allows me to very efficiently scan documents to my “digital file cabinet”. Here is a quick guide on how I do it!

What you will need:

1. An unused computer attached to your network

2. Google Desktop Search with network browsing enabled

3. A document scanner

4. A server based automatic OCR product

5. A file compression product ( optional but recommended )

Now to put it all together. How I have my system setup is an inexpensive desktop computer with Windows XP installed. Once all the applications are installed you don’t even need a monitor attached to this computer. The computer is visible on the network and has one folder shared the “File Cabinet” folder in my case. This computer is my stand alone digital file cabinet. Attached to it is a document scanner with a 30 page feeder. I have the scanner configured to scan to an “input” directory on the machine.

The automatic OCR processing product is configured to pick up images as soon as they arrive in the input folder “hot folder”, OCR them using specific index level OCR settings, and create a PDF with a hidden search-able layer. The resulting PDF is put into another hot folder that the PDF compression tool is watching. As soon as a PDF arrives in this folder it is instantly compressed and the compressed PDF is moved to the “File Cabinet Folder”.

Because Google desktop search is enabled to index all files in the “File Cabinet” folder the PDFs very quickly become a part of the index. Configure your Google desktop search to enable network searches so that any machine on the network can open a browser, go to a URL located on the digital file cabinet machine and be located with a search.

Once it’s setup it’s simply a matter of putting paper in the scanner and pressing the scan button, and you’re done. It’s that easy, and extremely useful!

Chris Riley – About