The ProZorro server has migrated to Ukraine—in itself a breakthrough for the domestic information technology industry—and the system continues to work regularly, proving public procurement reform to be one of the most successful in combating corruption in the country.

Since 2016, Ukraine has spent over Hr 1 trillion in public money on procurements—close to  $37 billion.

Using the ProZorro electronic procurement system has saved Ukrainian taxpayers more than Hr 55 billion in the last two years.

Although some pressure has been put on the system by politicians who claim to support reform while registering populists bills that contradict the public procurement development strategy approved by the Cabinet, in line with the European Uion-Ukraine Association Agreement, we can safely say that this reform has been successful, efficient and possibly the only reform that has actually been effective at reducing corruption.

It all started quite simply.

The ProZorro electronic procurement system was set up three years ago by an IT start-up based on subthreshold tenders, which were not regulated by law at that point. At the time, no one imagined that a start-up launched by a group of ambitious young people might become synonymous with change or that such changes might even be possible in Ukraine.

First, its effectiveness was demonstrated at a couple of ministries on a pilot basis.

Then, considerable effort went into extending this new system, whose name comes from the word “prozoro” or “transparently” in Ukrainian, to all government agencies in Ukraine.

Challenge #1: Setting up the system

When ProZorro was set up, Ukraine lacked the technologies and facilities to process, accumulate and store the immense quantities of open data that public procurement involves. The decision was made to use Amazon Web Services, the largest public cloud in the world. ProZorro’s presence on the Amazon server made it possible to concentrate on product development to make sure there would be no going back to paper procedures.

When the ProZorro system became mandatory for all government agencies in 2016, the question of a comprehensive information security system came up. Any such system had to ensure there would be no information leaks, no destruction or blocking of data, and no integrity breaches—all while providing access to open data.

According to Ukraine’s Law “On the Protection of Data Stored in Information and Telecommunications Systems” and the rules for the protection of data in information, telecommunication, and ICT systems approved by Cabinet Resolution #373 dated March 29, 2006, all ProZorro data had to be stored at a data center located in Ukraine. This meant starting the complex process of migration from Amazon Web Services to Ukraine.

Challenge #2: Migrating 50 terabytes + over 30 million files

The process of moving the ProZorro system took nine months and involved migrating all its IT landscapes from Amazon Web Services to the DeNovo G-Cloud in Ukraine.

Meanwhile, 50 terabytes of open data, over 30 million files and all the modules essential for the system to operate migrated unnoticed by the more than 200,000 ProZorro users. Since reliability is a key feature of the ProZorro system, ensuring uninterrupted operation was one of the biggest challenges of migration. ProZorro had to keep working in a way that 200,000 bidders would not suffer any inconveniences or question the system’s reliability.

To give an idea of the quantities involved: 50 terabytes of data is about like five Congressional libraries storing 35 million books, 68 million manuscripts, 1 million copies of newspapers spanning 3 centuries, 1 million U.S. federal government publications, and 5.3 million maps.

The unprecedented aspect of this migration was that, unlike international practice, where such a move usually happens in from a national storage location to Amazon, ProZorro was moving the other way. No one had done this before, so no one knew how it should to be done. One member of the ProZorro team joked that everything was basically clear—except for three little questions: Could it be done without shutting the system down even for a minute? How many people would be needed? And how much time would it take? Few people involved believed the task was do-able.

Migrating the ProZorro system was extremely complicated because it had to be done in a way that would not just maintain the information in the system, but also keep the entire system operational throughout the migration process. Every day, 3,000-5,000 calls for bids are announced, thousands of files are attached, and hundreds of offers are made in ProZorro.

This meant coming up with different technological solutions. When the decision to migrate was made, all the repositories in Ukraine were functioning only as backup servers, because ProZorro needs to update its data continuously. So, the migration team first set up a Ukrainian data center based on OpenStack Swift cloud storage, which was intended to store the system data. Next, special operation support drivers were developed, making it possible to save all the metadata in the documents stored on Amazon. Only then was the four-stage migration carried out, ensuring that users would not experience any issues with access to ProZorro.

Challenge #3: Post-migration support

Post-migration, ProZorro system support has become the key task. Providing top-notch service to users is the mission of the ProZorro and DeNovo teams: to safeguard identities and eliminate any issues related to system operation. At this time, the information is completely protected against loss using multiple backup support.

Since migration was completed in early March 2018, about 400,000 tenders worth a total of around Hr 300 billion have been announced in the ProZorro system.

Challenge #4: Ongoing data protection + streamlining

Today, the ProZorro team continues to work on establishing a comprehensive data protection system to guarantee that no information can be leaked, destroyed or blocked, that there can be no breach of integrity, and that the data access mode remains open. This process is expected to be completed by the end of 2018.

System development is a work-in-progress. ProZorro, the state-owned enterprise managed by Vasyl Zadvorniy, has already earned its keep and could invest in further developments to the entire system. The enterprise is now focusing primarily on improving efficiency for its customers and the suppliers of goods and services. Short-term plans include:

  • Setting up single-source procurement and e-catalogs. This will protect customers from bad faith suppliers and make the procurement process transparent and fair. For example, if a kindergarten or a school needs to buy stationery, they will no longer have to spend time preparing tender documentation and drafting specs. They can just go to a special online shop, where they specify their needs with a few clicks and choose from the list of transparently preselected suppliers who have passed the qualification-based selection process.
  • Working with open source data, integrating with state registers, and eliminating scanned paper documents. At present, ProZorro is already integrated with the State Auditing Service Register and the Register of Legal Entities and Individual Entrepreneurs. Integration with the State Fiscal Service and the Treasury to ensure the transparency of all payments made after the signing of agreements with suppliers is now underway. When completed, suppliers will be able to plan their cash flow and information will be obtained from government agencies in the simplest way possible.
  • Using artificial intelligence and machine learning. During some code fests with Amazon and other big data owners, algorithms were developed that make it possible to forecast the number of bidders with a probability of 72 percent. This means the customer will be able to know the possible number of suppliers in a particular tender right from the beginning, which should spur competition.

Why public procurement reform?

Ukraine spends Hr 1.5 billion daily on public procurement, a huge chunk of taxpayer money that has always been on the radar unscrupulous bureaucrats, elected officials and business owners. This has been particularly the case during election campaigns, such as the one that has already started in Ukraine. Moreover, public procurements are inevitably the focus of clashing interests, as a normal government wants to purchase supplies and services more cheaply, while business wants to sell for as high a price as possible. This can be a breeding ground for abuse.

ProZorro not only complies with the Law “On the Protection of Data Stored in Information and Telecommunications Systems,” but it also works reliably and remains one of the most successful reforms in Ukraine. The main accomplishment of the ProZorro system is that it is mandatory for all public agencies and it operates according to the principle, ” everyone sees everything.” ProZorro is able to control procurement and track whether public agencies are using taxpayer money efficiently and transparently: the more processes take place online, the more transparent they are.

For this reason, ProZorro has been made very simple: the system was developed in open source code and that makes it accessible to politicians and voters alike.

Using ProZorro tools may not help ascertain whether a certain politician has told the truth, but it will definitely determine the truth behind any procurement. Ukrainians can verify exactly what, where, how, and at what price their local school, city hospital or oblast administration made purchases using their taxes. It’s now possible to see whether a piano for the philharmonic hall, the asphalt for a highway, or beach-cleaning services were bought at an above-market price or not. What’s more, anyone can check this information with just a few clicks. This is ProZorro’s contribution to the consolidation and promotion of reforms. After all, ordinary Ukrainians really do care about where and how their hard-earned money is spent.

The back story

Migrating the ProZorro server has been a unique experience in Ukraine’s IT industry. It was made possible through the efforts of the ProZorro IT team led by Yevhen Yentis and specialists from the Quinta Group, MK Consulting and DeNovo, three Ukrainian companies.

In addition, $500,000 in funding from international donors covered the costs of the ProZorro migration: the United States Agency for International Development, UK Aid, the Eurasia Foundation  and the Transparency and Accountability in Public Administration and Services/TAPAS project.

Lesia Chmil is the deputy chief of party for U.S. Agency for International Development/UK Aid TAPAS, which stands for “transparency and accountability in public administration and services.”  Vasyl Zadvorniy, director general of ProZorro SO, contributed to this blog.