Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#1 2005-10-21 10:58:42

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

More advanced, or just smart search

Hi,

I’d really like to improve the search in textpattern. The version I’ve been dealing with is 4.0.1 (r888) behaves like this:

If i search for ‘toaster tire dean allen’ I only get hits in articles where those words are found in that particular order. So this means that if an article contains the following (very silly) sentence, I get a hit:

“While outside i noticed that toaster TiRE DeaN AlleN was happily jumping around with his wonderful dog Oliver”

Ofcourse the example above is very silly but I think that it illuminates my point fairly good.

Personally I’d like to get a hit on all articles containing any or all of the words that I search for. Ofcourse if one wanted to search for an entire sentence, you can achive this with quotes around the words: “toaster tire dean allen”.

Another thing related to the search is this:

One can, i MySQL, use special characters (>) to make certain words alittle more important in the search than others. Lets say that we assume that the first word we enter in our searchfield is the most important, the second is alittle less important and the third is the least important. Then we could make a search, with a single query to the database, that would consider the first word most important, the second slighty less important, and the last word the least important.

So if I search for ‘toaster tire dean allen’, and we say that we use this ‘smart’ importance thing above, we would get hits on article containing any or all of these words BUT the order that they are displayed is different: hits that MySQL considers most important are at the top.

This is abit complex but after having looked at the code used for searching in TXP today (or version 4.0.1 (r888) ) I believe that it could be implemented and I would gladly do it if this is the way for the future of Textpattern.

When I implement a site that will contain lots of content, then I will make a plugin using this type of search and see if it works ok.

Looks like my first day here at the forums could be a busy one ;)

Take care and happy TXP:ing!


.:8):.

Offline

#2 2005-10-21 11:14:49

Sencer
Archived Developer
From: cgn, de
Registered: 2004-03-23
Posts: 1,803
Website

Re: More advanced, or just smart search

Hi Henrik,

interesting ideas. Please consider that currently Textpattern works across mysql3, mysql4, and mysql5 versions; and for the next major release of TXP (no timeline exists yet) support for postgres is being looked at. Please keep these in mind, if you think about making a patch for us.

With the 4.0.x line we are more conservative with what goes in, so that existing sites that update are not negatively affected. I don’t know about potential side effects of what you suggest, but that may also be something to keep in mind.

Development-Related things are also discussed on these lists from time to time:
http://lists.textpattern.com/

You’ve probably already seen zem’s sticky in the other subforum about svn und patches:
http://forum.textpattern.com/viewtopic.php?id=6648

Welcome to Textpattern. ;)

Sencer

Offline

#3 2005-10-21 11:33:07

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

Re: More advanced, or just smart search

Thanks for the swift reply!

Valid points, valid points. My suggestion would be for MySQL 4.x and up (I am not sure if 3.23 is supported, I tried to find info about that on dev.mysql.com but wasn’t successful). One could ofcourse check the version of MySQL and only those that support this advanced way of searching would get it… but that would mean that txp-search would be different on certain platforms, which is not something we want…

I guess that in view of planned future support for other databases etc, I’d be better of making a plugin.

I wasn’t aware of the mail-list nor had I seen the post by zem about svn and patches, guess I’m a bad boy for not reading stickys before posting.

Thanks for the tips Sencer!

Take care folks!


.:8):.

Offline

#4 2005-11-12 02:06:16

Andrew
Plugin Author
Registered: 2004-02-23
Posts: 730

Re: More advanced, or just smart search

I am 100% for this form of searching. I track local searches on all Textpattern sites I am involved with, and the failure rate is extremely high for all – always because of the strict match searching.

Let’s take for instance my personal site, which has two rather lengthy articles relating to a silly hurricane tracking thing I made with Google Maps. Below is a recent sample of local site searches from the last week or two. All of the searches contain at least one word that exists within either article, yet the failure rate is at 75% (there are duplicates that I have omitted, so it’s actually closer to 80-85%):

  • hurricane maps
    • Found: Google Maps & hurricane data
  • hurricane plotting
    • Failure
  • hurricane maps
    • Found: Google hurricane maps update
  • hurricane ploting
    • Failure
  • hurricane trackings
    • Failure
  • hurricane tracking
    • Found: Google Maps & hurricane data
  • hurricane wilma
    • Failure
  • predicted maps of hurricane
    • Failure
  • predicted map of hurricane wilma
    • Failure
  • Hurricane Wilma
    • Failure
  • Hurricane tracking with google maps
    • Failure
  • hurricane andrew
    • Failure

I’m not sure what really needs to take place in order to fix this, but from a usability perspective this is bad. We cannot assume our users will enter single-word phrases into the search box. On the contrary, it’s been my experience that most searches nowadays consist of two or three words. On many websites, local search is the most important feature; it cannot be neglected.

So, Henrik, if you have the skills to patch this baby so that search actually works, the entire Textpattern community will be grateful.

Last edited by Andrew (2005-11-12 02:09:29)

Offline

#5 2005-11-12 08:25:03

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

Re: More advanced, or just smart search

Hi Andrew,

I’ve got a plugin in some sort of alpha state. I haven’t really been able to test it on a big site yet.

I’ll work on it some more (as soon as I can, when one is two ‘the girl’ don’t accept long hours infront of the computer) and I’ll let you test it and then you can tell me what you think of it, ok?

If anyone in the core-team, or what we should call the peeps that are responsible for the core development of TXP, needs help with this kind of search, just mail me and I’ll answer as many questions as I can. And those I don’t know the answer for, I’ll try to find out.

Take care and happy TXP-ing!


.:8):.

Offline

#6 2005-11-12 16:58:47

Andrew
Plugin Author
Registered: 2004-02-23
Posts: 730

Re: More advanced, or just smart search

Brilliant!

Offline

#7 2005-11-12 18:45:36

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

Re: More advanced, or just smart search

Hi Andrew,

you can find the TEST PLUGIN here: http://silviamedia.se/compooter/

one file is the plugin, the other is the form, get em both and I bet you will see which to install ;) …

Since its an early version there are loads of things for me to do before I release it (clean, optimize, add options, remove options etc …) but feel free to test it.

In the file with the form, you get an explanation of what tags you can use and what options or attributes exists.

Please get back to me and tell me what you think.

Happy TXP-ing!


.:8):.

Offline

#8 2005-11-12 19:08:07

Andrew
Plugin Author
Registered: 2004-02-23
Posts: 730

Re: More advanced, or just smart search

Are you planning on providing contextual information in results? For instance:

Search for “foo bar”

> Some Random Article
> blah blah blah blah foo blah blah blah bar

Last edited by Andrew (2005-11-12 19:43:56)

Offline

#9 2005-11-12 19:36:31

Andrew
Plugin Author
Registered: 2004-02-23
Posts: 730

Re: More advanced, or just smart search

Just out of curiosity, how does what you’ve made differ from ob1_advanced_search? I haven’t really dug in too deep as of yet.

I suppose for now a plugin is unavoidable, but at some point in the future expanded search options should be built into the core. I don’t have special needs when it comes to search and I really am not fond of the “advanced search” paradigm, because I rarely see average users actually use it.

The baseline support for search in Textpattern is what needs improving. If this plugin helps us reach that end, then I’m all for its development and incorporation into a patch. But really all I’m really looking for is exactly what exists now within Textpattern, just with more loose pattern searching.

Last edited by Andrew (2005-11-12 19:44:28)

Offline

#10 2005-11-12 19:57:51

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

Re: More advanced, or just smart search

The main difference between ob1_advanced_search is that mine lets you choose 3 different types of searches. And he use, for what I can see of his code, rlike and I dont, just the regular MATCH AGAINST.

I didn’t know about that plugin, I’m VERY new to txp and pluginmaking so my plug should be littered with silly misstakes and dumb things.

Infact I started making plugins before I even used textpattern (almost anyway).

My plugin uses about the same way of displaying the hits like the built in search in textpattern. That part is not perfect yet but I will improve it.

If you have a out-of-the-box installation of txp and add this plugin you will have a search that looks like the old search but searches differently.

The different searchoptions are just there for me, and you, to see what type of search works best and produces ‘the best’ result. Also, I thought, that since MySQL can provide these advanced search options, why not let the administrator choose what way is best for his/her site if that is what they want?

I agree that if this gets into the core there should not be alot of different search options that you have to go through to let the search work: it should work good out of the box and I guess that boolean search is the way to go. Perhaps you could also add the priority type search as standard aswell.

Test the plugin and get back to me with what you think: did it work well? Is it slow (it shouldn’t be for it should use index)? Does the result surprise you or is it what you thought it would be?

Basicly: what I have done now is a rough basic thing for testing this kind of search. If it works out fine, then I’ll take into consideration the feedback you, and perhaps others, have when I get this plugin released.

And if this is something that should go into the core of textpattern, then I think we need to discuss exactly how that will work or if we just use the plugin.

But thats not the issue here so lets forget it for a moment.

Keep txp-ing!

Last edited by Henrik Pejer (2005-11-12 19:59:48)


.:8):.

Offline

#11 2005-11-13 01:36:42

thebombsite
Archived Plugin Author
From: Exmouth, England
Registered: 2004-08-24
Posts: 3,251
Website

Re: More advanced, or just smart search

Hope you don’t mind Henrik but I’m trying this out on my test site at the moment. It seems to be working fine for me with one exception. In my search_results form template I have the <code><txp:search_result_excerpt /></code> but it doesn’t seem to be picking this up at all. I’m simply seeing … … where the excerpt should be.

Also it is picking up all sections, even those that are excluded from the search in the “Sections” tab.

Last edited by thebombsite (2005-11-13 01:45:27)


Stuart

In a Time of Universal Deceit
Telling the Truth is Revolutionary.

Offline

#12 2005-11-13 08:48:48

Henrik Pejer
Member
From: Sweden
Registered: 2005-10-13
Posts: 23

Re: More advanced, or just smart search

Its no problem if others than Andrew test it. The more that get the chance to try it out, the better.

Yeah, I know about the problem of it searching in ‘hidden’ areas. I just haven’t gotten around to fix that yet.

Good to know that search_result_excerpt doesn’t work, I’ll start looking into what search tags are available and make sure that they work when I continue with the plugin.

Last edited by Henrik Pejer (2005-11-13 12:19:13)


.:8):.

Offline

Board footer

Powered by FluxBB