search and archives - say it isnt so

[sound]

have tested but am not 100% sure yet
but does seems at first look that the archives are not included in the default forum search (sphinx)

you have to go to advanced search and then tick archive option

if the above is right then can i ask why has it been done this way

if there are separate searches for forum and archive then for my own uses this makes the whole archive feature unusable

please someone tell me that I am wong

[sound]

mmmm seems above is right

sphinx can search multiple indexes and so in theory could have a complete all in one forum/archive search, yes?
(and so in theory could have a complete site wide search)
but would guess there's no plans for this, or at least in the near future

so it looks like one of the major improvements is going to be unusable for anyone who needs to offer a decent search service that can search every past forum post with no fuss


all a bit disappointing really
do have to wonder yet again about invision logic when it comes to features like this

I mean why archive posts if the only way members can then find them is by then using the advanced search ?
how many casual members are going to be aware of that requirement when using the top search or indeed if even aware are then going to bother searching and viewing two sets of results ?

still have slight hope that have got this wrong and am missing something

[Rimi]

I would assume it's because the archived posts are moved to a different table (kind of the point) and that searching in multiple tables at once is a no no.

[sound]

I would assume it's because the archived posts are moved to a different table (kind of the point) and that searching in multiple tables at once is a no no.

sphinx apparently can search multiple indexes/tables

http://sphinxsearch.com/forum/view.html?id=6575



would think that chances are the ones who may have a lot of posts to archive may use sphinx

[Charles]

The point of the archive system is to take stress of your database by moving posts to a table that will be rarely queried. If every search was auto-queried to the archive table that sort of defeats the purpose :)

[sound]

The point of the archive system is to take stress of your database by moving posts to a table that will be rarely queried. If every search was auto-queried to the archive table that sort of defeats the purpose :smile:

removing large amounts of posts from the default search should not be the answer

do you seriously think people are going to do one search, browse the results and then find the advanced link and do another search and browse results all off their own back

would that work on this board ? are you going to archive posts on here then?
try it and see how it goes am sure having to double search in such an akward manner will go down a treat

the disapointing thing is that the basic sphinx setup to give the option to search two or one different tables is fairly easy to implement
in fact it could be said it is easier as there would be less duplication of code and you could still offer search 1 or 2 option

would it have hurt to think a bit about this before giving such an dismissive answer to my feedback

invision it seems has either missed or dropped the whole sphinx search more than one index aspect and by doing so have imo made a useful feature useless to anyone who runs a site that gets searched a lot for info/research etc

[bfarber]

Please keep in mind that sphinx is a secondary search method, so sometimes it can do more than we take advantage of.

This is something we can keep in mind for future versions in any event. :)

[sound]

Please keep in mind that sphinx is a secondary search method, so sometimes it can do more than we take advantage of.

This is something we can keep in mind for future versions in any event. :smile:

thanks for reply
fair enough, I think at times my requirements from the software must be a bit different than the out of the box solution
hence the frustration creeping in when that difference comes up
guess way to look at archive for myself is for content that doesn't require being made available for searching

[Charles]

If you want full functionality of topics then you should not archive them.

The archive system is designed as a sort of "deep freeze" for topics so normal day-to-day operations like search, reply, moderation actions, etc. are not available.

[h2ojunkie]

Being a forum admin of a 10,000,000 post site. The archive in theory is a great idea.

However, I have to agree with mikesound on this one.

The entire reason big sites run things like sphinx is to keep searching of a large database fast.

There is really no reason I can see to not take advantage of the power of sphinx and the archive feature together.

Sphinx can eaisly handle a single result across both the live and archive tables. I can't speak to using mysql full text search (since I stopped using that 9 million posts ago). But for a sphinx search setup, it should absolutly search the archive and live posts at the same time. There's just no downside to doing it that way.

[TSP]

If you want full functionality of topics then you should not archive them.

The archive system is designed as a sort of "deep freeze" for topics so normal day-to-day operations like search, reply, moderation actions, etc. are not available.

But Sphinx is not using a connection to the DB when searching, is it? I thought it made it's own indexes and copies of the post?

We're not using sphinx, but a custom made search that are using the Solr-engine. We export posts from the database to Solr for indexing and when searching it's only looked up to the information Solr have gathered. No DB-queries are used in the actual search. I would expect it to be similar for your built-in sphinx option?

As mentioned in the post above I can't see any harm in including the archive as part of the search results when using Sphinx. If you're having so many posts that you need to archive them, I wouldn't expect them to still be using the mysql full text search in the first place.

There is really no reason I can see to not take advantage of the power of sphinx and the archive feature together.

Indeed.

[Wolfie]

What about a system setting along with a front end option? System setting would toggle the front end feature to be available or not (and if on, minimum access required to use it.. guest, registered member, moderator, global moderator, admin) and that front end feature (when enabled) would let a member know that the search results don't include the archived content, with a link to do the same search but in the archived posts.

Would let the powers-that-be have the ability to do searches themselves while not letting regular members use it (save resources) or might just not let guests use it, or whatever else.

[bfarber]

Part of the problem is that there is almost no way to do this with built in MySQL search. While we could do it with Sphinx (because Sphinx can support multiple index searches), there is then a feature-set discrepancy between using sphinx and not using sphinx, which we try to minimize.

We will be taking a very hard look at the search system as a whole as we move to the Community Suite, and will consider specific improvements in closer detail then.

[sound]

Part of the problem is that there is almost no way to do this with built in MySQL search. While we could do it with Sphinx (because Sphinx can support multiple index searches), there is then a feature-set discrepancy between using sphinx and not using sphinx, which we try to minimize.

We will be taking a very hard look at the search system as a whole as we move to the Community Suite, and will consider specific improvements in closer detail then.

think said it before

my suggestion is to drop sphinx from the invision core

change it to a module that can be installed by those who wish to use it

back when sphinx was a stand alone module it was easy to modify/adapt
i had full site search with drupal intergrated before ipcontent

a seperate application would allow easier development and better features (all in one site search etc)
with sphinx

[Rimi]

you have to go to advanced search and then tick archive option

I don't see it anymore. >_>;

[Marcher Technologies]

think said it before

my suggestion is to drop sphinx from the invision core

change it to a module that can be installed by those who wish to use it

back when sphinx was a stand alone module it was easy to modify/adapt

i had full site search with drupal intergrated before ipcontent

a separate application would allow easier development and better features (all in one site search etc)

with sphinx

A separate application would mean less compatibility across not only the Suite, but any third-party application's supporting sphinx, sorry, cannot agree.

[Dmacleo]

one reason I went with IP was due to advertised compatibility with sphinx, needed the 2 letter and wildcard searches.

[sound]

A separate application would mean less compatibility across not only the Suite, but any third-party application's supporting sphinx, sorry, cannot agree.

sphinx has been slowly phased out - its just been dropped from the gallery
new content needs to be mysql if you wish instant listings
downloads next ?

soon could end up with it just being used for the forum search

if was an application third parties could just plug in to it

i would say with arguably less coding/dev than takes now

[Dmacleo]

just noticed it does not work on gallery now.
dammit.
maybe time to dump gallery.

[Marcher Technologies]

sphinx has been slowly phased out - its just been dropped from the gallery

new content needs to be mysql if you wish instant listings

downloads next ?

soon could end up with it just being used for the forum search

if was an application third parties could just plug in to it

i would say with arguably less coding/dev than takes now

It was dropped over a year ago(Gallery 4), and that is nothing new, and it *has* to be abstracted by application in the core, a 'global' site search is infeasible, what good is searching on downloads custom fields through forums indexes? None, it is an utterly wasteful act and you know it.

[sound]

It was dropped over a year ago(Gallery 4), and that is nothing new, and it *has* to be abstracted by application in the core, a 'global' site search is infeasible, what good is searching on downloads custom fields through forums indexes? None, it is an utterly wasteful act and you know it.

you want to tone down your words, if I thought it was a wasteful act i wouldn't suggest it


sphinx was integrated in gallery 4 - it has just been dropped from gallery version 5 this month- not a year ago
it didnt work out of the box unless you yourself fixed an error - which highlights imo the failing of the current integration

an application offering site search across all the current applications should be fairly easy to do
all you need to do with sphinx is give it a query and which index or indexes to search and it returns results

its what you do with the results that seems to be the hard part for some reason

I would have thought the benefits of what an 'across the whole site search' would bring are obvious to any forum owner and fits in with the whole community direction

[Marcher Technologies]

its what you do with the results that seems to be the hard part for some reason

It still has to format the results by application.

sphinx was integrated in gallery 4 - it has just been dropped from gallery version 5 this month- not a year ago

and..... no, don't state things without facts...
/gallery40002/admin/applications_addon/ips/gallery/extensions/search/engines/sphinx.php

<?php

/**

* <pre>

* Invision Power Services

* IP.Board v4.0.2

* Sphinx Gallery Search

* Last Updated: $Date: 2011-03-11 16:16:55 -0500 (Fri, 11 Mar 2011) $

* </pre>

*

* @author $Author: ips_terabyte $

* @copyright © 2001 - 2009 Invision Power Services, Inc.

* @license http://www.invisionpower.com/community/board/license.html

* @package IP.Board

* @subpackage Forums

* @link http://www.invisionpower.com

* @version $Rev: 8047 $

*/

if ( ! defined( 'IN_IPB' ) )

{

print "<h1>Incorrect access</h1>You cannot access this file directly. If you have recently upgraded, make sure you upgraded all the relevant files.";

exit();

}

include( IPS_ROOT_PATH . '/applications_addon/ips/gallery/extensions/search/engines/sql.php' );/*noLibHook*/

Straight from the client area, this is not new for this application, it has been proxying to sql for a while now.

[sound]

It still has to format the results by application.

and..... no, don't state things without facts...

/gallery40002/admin/applications_addon/ips/gallery/extensions/search/engines/sphinx.php

[CODE]

ent area, this is not new for this application, it has been proxying to sql for a while now.

yep ok, i stand corrected on the gallery 4 using sphinx
but the timing doesn't impact the fact that it was dropped from gallery as well as the other apps

using sphinx to get the results is the suggestion, the formating of the results ls secondary
sphinx will give owners the opportunity to list the results from all of the site features in a simple fashion and offer flexibility to tailor to suit each site
feel this would be best achived by a stand alone application for the reasons stated

I still feel it is a valid suggestion and hope its one that will be considered by invision

[Charles]

We're going to rethink searching with Suite 4.0. We agree it needs work.

[Wolfie]

Part of the problem is that there is almost no way to do this with built in MySQL search. While we could do it with Sphinx (because Sphinx can support multiple index searches), there is then a feature-set discrepancy between using sphinx and not using sphinx, which we try to minimize.

If you're referring at all to my suggestion, I said it with the idea of either two concurrent searches being done, or (better yet), have the search be performed on the archived content versus the main/unarchived/live content. But only when selected. Like a link saying, "Didn't find what you're looking for? Click here to search the archived content." or something similar.

my suggestion is to drop sphinx from the invision core

change it to a module that can be installed by those who wish to use it

A separate application would mean less compatibility across not only the Suite, but any third-party application's supporting sphinx, sorry, cannot agree.

I think mikesound might be onto something. A module to 'drop in', where you upload a file (or set of files) and then install/activate/configure/whatever within the ACP. The problem is that it would require redesigning the search engine altogether. Not a separate application really, but more like adding a 'plugin' to a game you play on your computer that gives you new features to use. To describe it a different way, similar to how you can add in new sign-in methods.

Not sure how feasible it would be, although it could open the door for 3rd party developers to come up with their own modules for site searches. Imagine someone making a module that is optimized for low traffic sites but isn't efficient on high volume sites (and vice-versa). Admin adds modules (and thus options) and then chooses which module to actively use (and with multiple installed, could simply change which to use at the drop of a hat, without having to download/install one since it'd already be there). Would make it easier to make 'plugin' modules that are optimized for sphinx or other server setups, etc.

you want to tone down your words, if I thought it was a wasteful act i wouldn't suggest it

<snip>

its what you do with the results that seems to be the hard part for some reason

He wasn't being rude/insulting (I know I didn't quote the post you were really referring to). He was just being blunt with what he was saying. Trust me, there's a difference.

We're going to rethink searching with Suite 4.0. We agree it needs work.

You know I think very highly of the developers. That aside, I think the search function will always need work. Because no matter what you do, there will always be a drawback to it and people will harp on those drawbacks. Look at Google, Yahoo, Bing, etc. I bet there are thousands (if not millions) of people who favor each of the search engines and will insist that all others give crappy results. Not a fault of the IPS developers, just how it is with searches overall.