Rebuilding Findability at Statistics Finland

New search waste of time and money or not? How we rebuilt findability at Statistics Finland Laura Kari, Statistics Finland Nordic Web Meeting 2025

Finding the future how and why we built a new search You might think search is boring. Or that it s on its way out, thanks to AI. Why build a search now? But a website without a search? That was also not an option This is the story of why we built a new search system at Statistics Finland and how we made choices that look forward Yes, AI is part of the future. But not all of it. 2 29 June, 2025 Statistics Finland

We did try AI but it wasnt enough (yet!) We made early PoCs with different AI vendors Demos are always promising, but PoCs with actual content were not They are usually a "black box" when it comes to data and used AI models At this point they were hard to trust and hard to verify But we understand that this all subject to change - anytime 3 29 June, 2025 Statistics Finland

What triggered the project? We have our old search on deprecated server o We were told in spring 2024 that the server that is hosting the search among many other things, will be deprecated, ie. closed in summer 2025. The current search is not well maintained, both technically and content-wise o Indexed data was not up-to-date. o The data was stored in on-premises servers without any possibility to use the data. o Maintenance and update responsibilities unclear. Old solution relied heavily on scraping an approach that is sometimes necessary, but it can make the system fragile, hard to control, and impossible to evolve 4 29 June, 2025 Statistics Finland

What users were experiencing? Fragmented content, inconsistent indexing Poor findability Unreliable results This all led to situation that the trust in our search was quite low, even our own experts didn't recommend using the search to our customers. 5 29 June, 2025 Statistics Finland

Ai will solve everything, right? "Why bother with the search soon nobody will use normal search anymore" - participant in first workshop - 6 29 June, 2025 Statistics Finland

So, why not just use AI? AI as a way to replace website search is not here yet o The current search, even with it's problems, is still in heavy use. yet, at least not totally reliably AI needs structured, high quality data Search is still useful fast and direct Content indexing is the foundation for both traditional search and AI based solutions But again, all this could change in less than a year, so anything that we build, needs to be as future-proof as it can 7 29 June, 2025 Statistics Finland

What if we started with the basics? Instead of a promise of a search interface that will change the world, we decided to focus on the building blocks and content, for once! 8 29 June, 2025 Statistics Finland

Strategic decision index first Focus on centralized, structured data Designed for search, APIs, and AI Used our own APIs that are already in place o CMS api, classification API, pxgraf API Built also a new API to get the content WE o Current PxWeb API does not return all the necessary metadata (text descriptions of various objects) that are vital for search index purposes. For example: content variable value descriptions (value note) that we want to index for each table o Also, the PxWeb API cannot handle the traffic of the scanning that the search indexing does o PxApi, open sourced, based on PxUtils WE need from the px-files 9 29 June, 2025 Statistics Finland

ElasticSearch Cloud was chosen to host the data Available as a managed service in Azure (no infrastructure maintenance) Close to an industry standard, at least in Finland o Has a good Finnish language support Can be used as a native data source for Azure OpenAI ElasticSearch has its own AI capabilities 10 29 June, 2025 Statistics Finland

We built something boring ...and that s the whole point. There is a search UI and no, it's not fancy 11 29 June, 2025 Statistics Finland

How we built it? Indexing application and search API in Azure o Everything is behind API management solution Structures sources, only little scraping left o Indexing data from different sources, mostly APIs, even some modernized scraping when necessary Keeping the content up-to-date o First option is always event-based indexing (index when things change) o Fallbacking to scheduled indexing Language agnostic and decoupled o Search API is available as REST API o Future proof for multiple clients and future tools 12 29 June, 2025 Statistics Finland

The search interface Simple, usable and clear Built for validation Will become official site search Although our focus was on indexing, we also built a straightforward search UI. It s intentionally minimal designed to validate the index and serve users with clarity. This same interface will soon replace our old search as the official solution on our site. 13 29 June, 2025 Statistics Finland

So... did it actually work or not? 14 29 June, 2025 Statistics Finland

What we've achieved? Built a working search system in just a few months Replaced fragile scraping (mostly) Unified access to all key content types (looking at you, Px-files!) Ready to be taken into production We realized that the success of the search functionality is not only technological, but has a lot to do with content We selected a product owner for search who will also be responsible for the overall SEO of our website, 15 29 June, 2025 Statistics Finland

What we've learned? Content must be indexed in in the the right right shape shape and at and at the the right right time time Content must be accessible first o This helps us in the search, our customers and AI bots Ranking Ranking results o Content experts are needed, but it's still hard :( results is is hard hard A modern search can be built in a a matter matter of of months months 16 29 June, 2025 Statistics Finland

Search can be dead... ...eventually 17 29 June, 2025 Statistics Finland

What's next the role of AI? Structured content enables future AI use Our index could power an OpenAI-based assistant We can really easily start preparing the data and indexing for RAG-style use cases But we are not rushing let's get this machine running first! 18 29 June, 2025 Statistics Finland

Now is your turn! What did this make you think about? Is your search ready for the future? Is your content indexed and maintained the right way? Who decides what's important in your search results? What role does AI play in your services? 19 29 June, 2025 Statistics Finland