Skip to main content

Posts

Showing posts from 2018

Moving 21 million documents to Microsoft Azure with Spring Batch

Background While updating a messaging system we decided to move all the unstructured content out of a relational database to a dedicated document store. Keeping only an Identifier for a document considerably reduces the size of the database thus making savings in storage and facilitating the database administration work. Driven by criteria of ease of management, security and cost, the chosen document store was a Software as a service solution: SharePoint online which runs on the Microsoft cloud Azure . Problem to solve Very briefly: move 21 million documents from our in premise to the cloud in less than 30 days . In more detail As for all migrations we need to move the data and map the source identifier and the ID in the new location. This is to allow references to the migrated to be updated. The stages high level for this move were ... Scan the eligible items to be migrated and store their identification and details to access their content in a single table. This table a