newzealand.govt.nz using Google custom search (was NZ govt feed standard updated)

Hi Tim
 
I'll start with some background on how the search solution we have implemented
works, and how we manage it. There is no secret we use Google Site Search
which gives us programmatic access to the Google index. This index already
exists, we take a slice by restricting it to the government web space. This is
done at the URL level. It's important to remember the government webspace is
very large and encompasses a whole lot more than .govt.nz sites and includes a
stack of other .co.nz, .org.nz, .com, .cri and .mil to name a few. This is
huge number of pages and documents, 5 million is our estimate so having access
to an index that already exists and covers all those documents gives us a
brilliant starting base.
 
We manage the list of government websites (some die on the vine, and new ones
are created so it is an evolving and changing list)
 
We pass search queries through to the API, XML is returned and we format that
in two ways, one as HTML (did you notice the site is HTML5?), and also as ATOM
as per the new government standard.
 
We lightly massage the way the search results are returned. Some of the
massaging includes:
 
Using synonyms to bridge government language with real language, an example is
here, where you can see a search on anti smacking refined by acts of parliament
returns the relevant act, even though the term smacking is not actually used in
any legislation:
 
http://newzealand.govt​.nz​/search​?format​=html​&​;query​=anti​+smacking​&​;refine​[0]
 
We have a few spotlights to bring some relevant information to the top of the
search results, an example was with the Canterbury earthquake we 'spotlighted'
sites which were useful to people. We clearly identify these via formatting.
 
http://newzealand.govt​.nz​/search​?query​=earthquake
http://newzealand.govt​.nz​/search​?query​=john​+key​&​;s​=Search
 
We have 'facets' or 'refinements' which allow a search to be filtered based on
a subset of the index we define. We do this based on URL patterns. The news
search is across defined news pages on government websites. We identify these
from feeds and pages (but they are all URLs with patterns at the end of the
day).
 
Adding facets, synonyms and managing the index is now trivial to low effort,
for us to create a facet, its pretty straightforward, so we can be more
responsive and agile.
 
One thing is the closeness to the index allows us to use search as a tool to
solve problems and allow more relevant information to be surfaced.
 
*Is it possible to make the parameters you have given to Google explicit on the
website?
 
I'm not entirely sure exactly what you mean here, so if I am ambiguous, ask
again. All parameters we pass through to Google can be seen in the URL.
There is actually a stack we can pass through - which is what I mean by being
able to use search as a tool. You can append the URL if you want to play with
it, here is a list of additional filters that can be used. You might notice
with the news "read more" link on the front page we append the parameter
as_qdr=w, this returns results within the past week, we could used something
like as_qdr=3d to return the past three days, this is trivial for us to change
now.
 
We have a whole host of request parameters that can be use if the use case
presents itself. Most are documented here:
 
http://www.google.com​/cse​/docs​/resultsxml​.html​#wsRequestParameters
 
 
 
*Is there any advantage of using nz.govt.nz for government content over
Google's own index?
 
As Google does not know what is government and what is not then, I'd have to
say yes. Searching for acts of parliament, or open datasets is a whole lot
trickier in Google. Search is a big part of the site but we're not a search
engine, search is one way for people to find the relevant government
information when they are on the site, other ways include the A-Z and the
Browse.
 
*I'm interested in knowing how DIA has prioritised the results.
 
We're not really prioritising the results, we don't say one result is better
than another, this is handled organically by the search algorithms, we take
what the index throws back at us, then we do a wee bit of massaging with
refinements, and synonyms.
 
*130 results
We'll need to rewrite the pager to display more than 130 results, currently
about 5% of people go past the first results page, and that is to page 2. If
you look at the URL and play with the parameters you can go past the 130
results. The diminishing value of results further down the index make this a
lower priority task at present.

Thanks for the detailed information here Jaime. Thanks for taking the time
to draft it. I think it would be very beneficial to have this email
formatted for the site and included in a page like "How we conduct search"
 
On 17 October 2010 10:33, Jamie <<email obscured>> wrote:
 
> Hi Tim
>
> I'll start with some background on how the search solution we have
> implemented works, and how we manage it. There is no secret we use Google
> Site Search [...]
>
 
I think it's really good that you're leveraging off of existing, cheap,
technologies.
 
 
> We lightly massage the way the search results are returned. Some of the
> massaging includes:
>
> Using synonyms to bridge government language with real language, an example
> is here, where you can see a search on anti smacking refined by acts of
> parliament returns the relevant act, even though the term smacking is not
> actually used in any legislation:
>
>
>
http://newzealand.govt​.nz​/search​?format​=html​&​;query​=anti​+smacking​&​;refine​[0]
>
>
This goes far beyond stemming words, and so forth. Well done. Is this
massaging done via a machine learning process? I'm assuming they're curated
by hand.
 
 
> We have a few spotlights to bring some relevant information to the top of
> the search results, an example was with the Canterbury earthquake we
> 'spotlighted' sites which were useful to people. We clearly identify these
> via formatting.
>
> http://newzealand.govt​.nz​/search​?query​=earthquake
> http://newzealand.govt​.nz​/search​?query​=john​+key​&​;s​=Search
> [...]
>
*Is it possible to make the parameters you have given to Google explicit on
> the website?
>
> I'm not entirely sure exactly what you mean here, so if I am ambiguous, ask
> again. [...]
>
 
At the time when I wrote that email, I thought what you were doing was much
closer to a generic custom search. That is, you just tell Google which
keywords to add or remove from the custom search. It looks like you're
providing a much richer service. I haven't used Custom Search too much
myself, but it seems the parameters are far richer than I had anticipated.
 
*Is there any advantage of using nz.govt.nz for government content over
> Google's own index?
>
> [...] Search is a big part of the site but we're not a search engine,
> search is one way for people to find the relevant government information
> when they are on the site, other ways include the A-Z and the Browse.
>
 
 
*130 results
> We'll need to rewrite the pager to display more than 130 results, currently
> about 5% of people go past the first results page, [...]
>
 
Agreed, it seems silly to change how the pagination etc works for that level
of use. Maybe you could indicate that there are "more than 130" results.
That would be a single line of code somewhere.
 
HTML5:
 
Although changing adding <!DOCTYPE html> does make things HTML5, there are
several quick alterations that will make things smoother.
 
If I understand the hype correctly, if you change the HTML to include
semantic tags, then you will be able to remove accessibility features like
"Skip to content", the user agent will be able to render the content first.
There are lots of possibilities of extending the site to make things more
personal and relevant, especially with local storage.
 
Thanks again Jaime for taking the time to respond on a weekend.

Tim McNamara wrote:
 
> This goes far beyond stemming words, and so forth. Well done. Is this
> massaging done via a machine learning process? I'm assuming they're curated
> by hand.
 
In fact, the portal (as it was then called) has done this since the version
that was launched 8 years ago in 2010. Then, a lot of work was put into
constructing two thesauruses, one about "subjects" (i.e. nouns) and the other
about "functions", i.e. verbs. The aim was to find "canine registration" when
some typed "dog licensing".
 
Cheers
 
Colin

Sure, I assumed it had been replaced. My point was that it's not new to do
semantic analysis of search terms in govt search.
 
Cheers
 
Colin
 
 
On 19/10/2010, at 10:48 AM, Nadia Webster wrote:
 
> Hmmm - sorry Colin - that isn't correct anymore. Hand-crafted metadata
(which used the thesauri) proved to have a costly overhead. Developments in
search meant that since June 2007 we've been using the .govt.nz slice of a
public index plus a search integrator. This means we can leverage off the R&D
of large companies and don't have the expense of indexing sites that are
already indexed.
>
> Now we're using the synonyms feature in the Google Custom Site Search and can
update based on web search statistics.