[FX.php List] To data condition or not data condition. That is the question

[Dale Bengston]

Hi Jonathan,

I think you've answered your own question. It doesn't make sense to condition data that's going to be filtered out to such an extent. I can appreciate your desire to clean it up as much as possible as a courtesy to the pairs of eyes doing the reviews.

If this were my project, I'd probably stop at doing basic JavaScript validation on emails and phone numbers and postal codes and the like. You can also make your best attempt with on-screen instructions to get people to type using proper case and syntax (for what it's worth). Then the focus becomes managing the expectations of the reviewers, and establishing editorial guidelines so the humans are all following the same rules. Nothing worse than your reviewers spending time re-editing records over and over because one prefers St and another prefers St. and another writes out Street. Having editorial guidelines starts with the up-front assumption that the reviewers will be doing the data conditioning, which avoids heartache when the site goes live.

Dale

On Dec 6, 2009, at 10:01 AM, Jonathan Schwartz wrote:

> You see, that is the reason not to even start with conditioning if 95% the data is not going to flow upstream.  For the 5% that will flow upstream, only a small portion of those will need repair, so it boils down to not even starting.
> 
> I like your "Mc" tool below, but then we have all the other exceptions:
> 	deGalle
> 	d'Angello
> 	de la Cruz
> 	and on...
> 
> In this project, better not to data condition at all.  I have my answer.
> 
> Thx
> 
> Jonathan
> 
> 
> 
>> If you are to make proper() on names you would probably have to make
>> exceptions...
>> 
>> McDuck
>> 
>> case( left( $lastname; 2 ) = "mc" ; proper( left( $lastname; 2 ) ) &
>> proper( middle( $lastname; 3; -1 ) );
>> /* misc other exceptions */
>> proper( $lastname )
>> )
>> 
>> ggt
>> 
>> 2009/12/5 Jonathan Schwartz <jschwartz at exit445.com>:
>>> Hi Folks,
>>> 
>>> To data condition or not data condition. That is the question
>>> 
>>> This is a philosophical question, I believe.
>>> 
>>> Up until now, most of my projects have dealt with collecting records where
>>> 100% of the records would be used later in the client's FileMaker systems.
>>> It made sense to apply capitalization and phone number formatting and such
>>> to have the data comply with existing data standards.
>>> 
>>> In a new project, however, records are being submitted that will be stored
>>> in a remote database and undergo a series of reviews before potentially
>>> being accepted and copied to the main database.  Perhaps 5% will make it.
>>> 
>>> Because..
>>>        - Data conditioning isn't a perfect process
>>>        - It costs time to develop
>>>        - It costs cycles to run
>>>        - It "taints" the original data entry. (What would ee cummings think
>>> about having his name be capitalized?)
>>>        - Only a small number of records will emerge from the process
>>> 
>>> ...does it makes sense to perform this step?
>>> 
>>> Also, is this a question best left to the client?
>>> 
>>> Would appreciate the insight of folks who have dealt with this....except Bob
>>> of course. ;-)
>>> 
>>> Jonathan
>>> --
>>> Jonathan Schwartz
>>> Exit 445 Group
>>> jonathan at exit445.com
>>> http://www.exit445.com
>>> 415-370-5011
>>> _______________________________________________
>>> FX.php_List mailing list
>>> FX.php_List at mail.iviking.org
>>> http://www.iviking.org/mailman/listinfo/fx.php_list
>>> 
>> _______________________________________________
>> FX.php_List mailing list
>> FX.php_List at mail.iviking.org
>> http://www.iviking.org/mailman/listinfo/fx.php_list
> 
> 
> -- 
> Jonathan Schwartz
> Exit 445 Group
> jonathan at exit445.com
> http://www.exit445.com
> 415-370-5011
> _______________________________________________
> FX.php_List mailing list
> FX.php_List at mail.iviking.org
> http://www.iviking.org/mailman/listinfo/fx.php_list