Why ext/filter?

Friday, December 22. 2006

From time to time the ext/filter topic comes up. Yesterday one of its authors was attacking serendipity for having code inside that bypasses ext/filter to get the original RAW values. Quite amusing how he claims the only correct way to do input filtering is to use the ext/filter functions and only use your own functions as a fallback. Infact the only sane way is the other way around: Have your own input filtering functions and do not even think about using ext/filter. There was actually never a need for ext/filter because everything it does can be implemented on a PHP level with plain PHP functions. ext/filter is just a new irritating API to already existing functionality, that potentially introduces new security holes, because it reimplements existing stuff again.

ext/filter is a similar stupid idea like magic_quotes_gpc. When you use it, the security of your application depends on the server configuration and no longer on your application code. And if you have similar input filtering functions to be compatible with servers that do not have ext/filter (which is the majority of all servers) then there is no reason at all to use ext/filter. Supporting ext/filter makes your code more complex and in case of a bug in ext/filter you have no chance to protect your users, because security only relies on the admin's will to upgrade. Considering the fact, that ext/filter is a builtin extension this means he has to recompile PHP.

Additionally ext/filter will keep a copy of all GPC variables in memory even when it is not used, therefore it is wise to add --disable-filter to your configure line if you do not want to waste ressources.

Posted by Stefan Esser in Security, PHP at 12:35

Comments (21)

Link to this post

Comments

Display comments as (Linear | Threaded)

#1 Pierre wrote (Link) ()

"its authors was attacking serendipity"

First, I was alone to talk about this problem. Secondly I was not attacking anyone or projects but gave an advice in a better way.

And you are completely wrong to tell that ext/filter implementation is possible in user land. In what does it make the code more complex? It simplifies it.

About the rest of your text, I see that nothing will ever change in you. You keep insult 99% of the PHP people's work and tell that the 50% of the rest are braindead.

posted on Friday, December 22. 2006

#1.1 Stefan Esser wrote (Link) ()

First Pierre,

please learn to quote:

"its authors was attacking serendipity"

has a completely different meaning than

"one of its authors..."

If you quote correctly your first paragraph is bogus. Additionally your advice is just your personal opinion. It is not the better way. The better way is having input filtering in your application code, because that makes it more portable/compatible. No need for ext/filter at all.

Secondly supporting ext/filter makes the code more complex, because a compatible and portable application will always need the same input filtering done in user space for servers where ext/filter is unavailable. This means more complex code.

Pierre, one advice. Please grow up. Noone here is eating your "50% braindead" arguments. Try to give any technical argument that renders one of my statements wrong. Will be pretty hard because I stated pure facts, like I always do. Or do you want to hide the fact that your elite space trimming code in ext/filter is resonsible for remote code execution exploits against applications using ext/filter?

posted on Friday, December 22. 2006

#1.1.1 Anonymous Bastard ()

Even though I agree with you regarding some aspects of ext/filter, I think attacking all the other people on a personal level doesn't help your mission (even if it's sometimes funny). So please, even if the people you're talking to are bastards or dumb sheeps, don't tell 'em. Because they will only hear that and not your legitimate complaints. I assume you know Hans Reiser and his problems with the Linux developers when it comes to the inclusion of ReiserFS. It always ends with a giant flame war and lots of personal attacks. The result is, that Hans Reiser may feel better, but Reiser 4 still isn't part of the mainline kernel.

Or to say it short: Stefan, you're an authority when it comes to security. And you're doing a great job. But you'll loose your status if you always say what you're thinking about other persons. I had to learn that too. And it wasn't easy. But it's worth.

posted on Friday, December 22. 2006

#2 Christopher Kunz wrote (Link) ()

A good example for one of the authors' attitude towards known bugs is seen here: http://devzone.zend.com/node/view/id/1113 (last comment). The bug was open for a month, and obviously acknowledged - and yet the only comment for this was "pah, use your own regular expression". That really raises the question: Why use ext/filter in the first place?
Thanks to Derick, the problem was solved within minutes after alerting him to the issue, but the larger issue itself persists: I'll have to recompile PHP for a more usable e-mail filter expression. And I'll have a weird feeling that at least one of the main advocates for ext/filter maybe actively refuses to fix bugs.
There's still a ton of other issues, but I am confident I'll be able to sort them out with a longish mail to the authors.

posted on Friday, December 22. 2006

#3 A.Sleep wrote (Link) ()

I agree 100%. While ext/filter is a nifty idea, I won't use it due to the compatibility issues from one server to the next and the potential issues with debugging. What ext/filter did for me (prior to it being ext/filter) was provide a helpful framework for my own filtering.

posted on Friday, December 22. 2006

#4 Pierre ()

"A good example for one of the authors' attitude towards known bugs is seen here: http://devzone.zend.com/node/view/id/1113 (last comment)."

Blogs and web sites are in no way bug reports. You know it.

" The bug was open for a month, and obviously acknowledged - and yet the only comment for this was "pah, use your own regular expression"."

That was not my only answer.

"And I'll have a weird feeling that at least one of the main advocates for ext/filter maybe actively refuses to fix bugs."

Can you stop to spread FUDs and insult our work. Tell me when I refuse to fix bugs or reject issues. The only persons who reject to report bugs were on your side. Bringing insults and politics to technical discussions (check internals and cvs archive, I asked numerous time to report bugs).

By the way, Christopher, we are still waiting your "deep review" (http://www.christopher-kunz.de/archives/116-Thoughts-on-extfilter-et-al..html).)

posted on Friday, December 22. 2006

#4.1 Stefan Esser wrote (Link) ()

First of all Pierre,

we do not exist to fix all faults in PHP, because otherwise we would have no time to do anything else.

Secondly we have reported several security bugs in ext/filter like filter bypass and remote code execution to security@php.net

Maybe you try to get your work right, so that we do not need to write anything about your stuff.

With your kind of attitude you can be happy that we report any bug at all and do not sell them to some guys very interested in PHP exploits.

If the PHP developers ever grow up and loose their arrogant behaviour, they can expect a different treatment.

posted on Friday, December 22. 2006

#4.1.1 Christopher Kunz wrote (Link) ()

"If the PHP developers ever grow up and loose their arrogant behaviour, they can expect a different treatment."

I think in this case you absolutely *HAVE* to refer to a singular, not the plural.

posted on Friday, December 22. 2006

#4.2 Christopher Kunz wrote (Link) ()

You can rest assured, the appropriate persons will get a "deep review" - in fact some of my remarks have already been acknowledged by people that are actually willing to fix stuff instead of imposing bugfixing onto the end user.
I find your comment at the very least insightful. I think it tells more about your attitude than you were intending to show.

posted on Friday, December 22. 2006

#5 Ilia Alshanetsky wrote (Link) ()

A number of valid points were raised here, so as one of the filter authors allow me to set the record straight

.

First all, in my opinion if your application has an established input filtering/processing mechanism that you feel is solid, there is really no reason to use the filter. And a solution applied by Serendipity developers is perfectly acceptable. My only recommendation would be that they first try to use .htaccess to clear out the default_filter setting or perhaps during install require the user to do so themselves.

Stefan is correct in saying that everything the filter can do, can be done using native PHP functions. However, it means that most developers will end up cooking up their own filtering mechanism often repeating the same common filters time and time again. The filter extension tries to provide a common interface that includes most commonly used data filters/validators. Which means you don't need to worry about writing,testing,securing the code.

Unfortunately, filter in 5.2.0 was not bug free and now in CVS there are over a dozen fixes to the code. And I happy to say that to my knowledge there are no open, unresolved bugs in PHP bug system. That is not to say that other bugs may not exist, it would be exceptionally arrogant to claim code is bug free. If you do know a bug please report it via bugs.php.net. Posting bugs on forums is not helping anyone because if the developers don't read them, how can we know about and possibly solve them. So, please if you have a bug or think it is a bug, repot it on the bug system.

posted on Friday, December 22. 2006

#5.1 Anonymous ()

Ilia said:
"...That is not to say that other bugs may not exist, it would be exceptionally arrogant to claim code is bug free..."

Sorry, Ilia. In real life - let's better say in professional business software companies - zero defect state is crucial for both customer and software manufacturer. In last consequence, the customer expects software as *bug free* - and that is what reliable software is when it is being released according to customers specs or expectations. If PHP developers would just little more shift their point of view from that "we develop open source and therefore noone should trust in our code" or even that old "use at your own risk" attitude towards a more professional and market-focused opinion, they'll surely earn more respect, more customers, more supporters and more self-esteem.

posted on Monday, December 25. 2006

#5.1.1 Lars Pohlmann wrote (Link) ()

There is no such thing as *bug-free*, especially not in real-(business)-life, when the complexity exceeds "Hello World" by far.
And companies would be far better off accepting that and putting more effort into developing strategies *how* to handle occuring bugs, instead of a "it-mustn't-be-so-it-can't-be" mentality. But that just as a sidenote.

posted on Thursday, December 28. 2006

#6 Anonymous ()

Is there a significant difference in performance between using included filters in PHP or creating functions that handle sanitation?

This would be ignoring portability or security benefits of either approach, of course

posted on Friday, December 22. 2006

#6.1 Stefan Esser wrote (Link) ()

In theory userspace filter are slower than C functions.
(I say in theory, because I did not perform a benchmark on ext/filter)

However when you look at the situation from a realistic point of view, then the amount of input variables is not very big. So even if your filtering in userspace is 10 times slower because this only happens at the beginning of the script for a few request variables it should not make that big a difference in total.

posted on Friday, December 22. 2006

#6.1.1 Ilia Alshanetsky wrote (Link) ()

Unless there is a bug or bad code C functions will always be faster then PHP functions, that's the nature of the game. Stefan is correct in saying that it most cases it wouldn't matter, if you process a lot of data (due to large individual request or many small requests) any little overhead begins to add up. This may make no difference to someone running a small site, but it makes a world of difference to web hosting companies and large sites where every bit of performance counts.

posted on Friday, December 22. 2006

#7 Lukas wrote (Link) ()

I have not really begun using ext/filter, but one of the key idea that Rasmus had when he created the hooks for something like ext/filter was exactly to have the ability to have central configuration of your filters. This allows Yahoo! to basically take a 3rd party app, shut all input out entirely and then start poking holes until the app works again. Apparently this is making it much easier for them to audit7secure 3rd party code.

posted on Friday, December 22. 2006

#7.1 Rasmus ()

Yes, ext/filter is more geared towards implementing a centralized input filter policy on a server and yes, even though Stefan thinks this is a negative thing, he describes it perfectly:

"the security of your application depends on the server configuration
and no longer on your application code"

I don't trust your code, sorry. I don't really care who you are or what sort of wonderful application you have written. I don't trust it and as such I am going to turn on a very strict input filter and feed everything through that. If something in your application breaks, then I'll fix it myself by putting in as strict a filter as possible on that particular input variable that still allows the application to work. The idea here is to error on the safe side. I'd rather have a slightly broken application than an insecure application running on my server.

And as Lukas mentions, it is also rather easy to audit an application to see if if it pokes any holes in the data firewall simply by grepping for any instances of filter_get or more importantly any instances of FILTER_UNSAFE_RAW and the having a look at what the app does with that data instead of going through every single input variable. Of course, if an app loops through every input variable and fetches the raw version, then that is the first thing that gets removed before the app is deployed.

posted on Saturday, December 23. 2006

#7.1.1 Lars Pohlmann wrote (Link) ()

Rasmus has a point here. There are two approaches to the input-filtering (I never worked with ext/filter, so maybe I'm wrong about some details):

1.) You're a developer of an application, that needs to run on various server-configuration, hence you can not trust the server to have filtering in place. You need to do your own input-filtering (which you anyway should always do).

2.) You develop applications which are supposed to run in an enviroment you have control over, and furthermore you develop in a team, where not everybody is on the same level of experience.
In this case I'd prefer to have a server-based solution, which filters all
input and lets me know if something breaks in doing so.

In short: Input-filtering _should_ be done in the application, but when you are responsible for the security, and have many developers in your
team, and not the time to review _all_ the code which is getting into the wild, a server-based solution is a "nice-to-have".

posted on Wednesday, December 27. 2006

#8 Robert wrote (Link) ()

"The filter extension tries to provide a common interface that includes most commonly used data filters/validators. Which means you don't need to worry about writing,testing,securing the code."
-- Ilia (Comment 5)

What about the less commonly used filters? Shouldn't developers always be concerned with input validation/sanitization, especially when using a tool that they have not written themselves? I am not asking to be a devil's advocate. I am asking this sincerely as I subscribe to the notion that I am solely responsible for my code and what it does. If a client's site get's hacked and it is because of a flaw in the language the site was coded in, then it is my fault for not knowing my language. I don't think the client would consider firing PHP, so...

"The idea here is to error on the safe side."
-- Rasmus (Comment 7.1)

If not all systems are the same and not all implementations of PHP are the same and not all server setups are the same then wouldn't it stand to reason that input/output management be handled within the application rather than with language tools that implement "more commonly used input filters" (as opposed to as many as possible)? Or am I just being silly?

posted on Sunday, December 24. 2006

#9 Paul Gregg wrote (Link) ()

Rasmus has a point that ext/filter is useful *at the server level*. However, all promotion of ext/filter that I have seen is that it is aimed at developers to use in their own application. This is a wholly different use-case and requires two different discussions.

For serverside use, such as Rasmus described, it puts you - the server admin - in charge of the input. I have not looked at how this can be implemented (perhaps handled before php script execution in the manner of file uploads) - but any *application* use of ext/filter is utterly irrelevant.

For application use, I understand Stefan's point - and I believe this is the context that he is discussing. I agree with many of Stefan's points and regulars on efnet/#php will be aware of my objections on that front. The email filter in ext/filter is seriously broken and validates emails such as 'a@-b.com' '-@..com' 'invalid@special.museum-' yet rejects whitespaced ones like ' a@bar.com ' (which is valid btw).

Paul.

posted on Monday, April 16. 2007

#10 Gregory Boshoff ()

If your only passing around basic types such as integers or alphanumeric strings. I suspect that you would be better off using PHP ctype ext C functions to vaildate input vars rather than using PHP's Filter Functions.

Why aren't these extensions already part of the core?

posted on Monday, November 5. 2007

Add Comment

Name:

Email:

Homepage:

In reply to:

Comment:
Standard emoticons like :-) and ;-) are converted to images.

Remember Information? Subscribe to this entry
Submitted comments will be subject to moderation before being displayed.

Calendar