Q:Hello Alex. Regarding your article on re-indexin, transactions and locks, could you provide an up-to-date feedback on whether it is safe to remove the foreign key that would give you locking errors. In my client's setup, we have 30 store views (i know that is a lot, but the client just won't accept my advice and have some of them removed. We have also installed Mirasvit Async Indexing for re-indexing in the background. Thank you for your time.
I think indexes on those tables are not completely necessary, as they the tables they reference (the index names) are not going to change. Therefore, removing the index does not affect the correctness of the application, and not having to lock those rows improves the reindexing considerably.
That said, I only used this technique with Magento 1.7, so things might have changed in the last version. And have never used the Mirasvit Async Indexing, but I guess it would not miss those indexes anyway.
SRP on Rails: Form Objects (part 2)
This is the second in a series of posts about Form objects. The first one can be found here.
Ok, for our Events application, we got the user to create the event with it’s most basic information. Now we want them to be able to add the details.
They need to add information about the price of the ticket, and about the Venue. The price of the tickets will be a part of the Event. The Venue will have an address, and a short description. How do we edit it?
Option 1: All in the same object
Maybe, for simplicity, you add all the fields in the Event. It’s just two extra fields, right? Well, that’s certainly an option. If you are not going to pay much attention to the venues, then maybe that’s the way to go. If you’re going to use the venues (for instance, reuse them between events), then they should be a separate object.
Option 2: #accepts_nested_attributes_for:
Rails has a great tool in it’s belt for saving multiple objects at a time: #accepts_nested_attributes_for:. This works relatively simply, you just embed your objects in the form using #fields_for and you use _attributes from the controller side.
<%= form_for :event do %>
<%= input_tag :price %>
<%= fields_for :venue do %>
<%= input_tag :address %>
<%= input_tag :description %>
<% end %>
<% end %>
That will sent to the browser this hash:
{event:
{id: 1,
price: '50',
venue_attributes: {
address: '308 Negra Arroyo lane',
description: 'The house with the pizza in the roof' }}}
The only requirement you have in your model, is to add:
class Event < ActiveRecord::Base belongs_to :venue accepts_nested_attributes_for :venue end
It’s oh so simple. The venue belongs to the event. The concept of composition is clear in the form, and in the controller, and Rails uses that to simplify a lot of work.
This, however, leads down a dangerous path. The Event, from now on, is responsible for managing the Venue. It’s a lot of responsibility that we would like to keep separate. Why is that? For one, we don’t need the Event to have the burden of managing the Venue all the time. That forces the programmer working on the system to think about the Venue every time he’s working with the Event. And also, any changes in Venue might impact Event, which is a class that’s going to be very important for our code base, so we should try to keep it as clean as possible.
Option 3: Form Object to the rescue
What we can do, is create a new Form Object. This one will handle the updating of the Details in the Event.
You should note that we’re not changing the original form object, but creating a new one. We do that so we can keep the two logics separated.
class EventDetailsForm < Syrup::FormObject
wraps :event
accepts_nested_attributes_for :venue
form_name :event_details
def after_find(params)
event.build_venue if event.venue.nil?
end
end
As you can see, the logic is much more clear, as only the relevant parts of it are in this class and no logic regarding the other controller and form is available.
We set a form_name so that the routes generated are more user friendly (like /events_details/4/edit). And the #after_find method deals with the case of an event not having yet a venue, which then creates an empty one.
Conclusion
As you can see, Form Objects are a good way of organizing the logic in your application, maintaining part of the logic that does not belong neither in the model or the controller in a separate entity.
SRP on Rails: Form Objects
Rails tools for creating forms and REST controllers are great, easy, and get the job done in most cases, and with very little effort from our side.
Editing the logged in users name, creating a new address, etc, are good examples of actions that can be done automatically. The view shows the fields, pre-populates the data and then with the controller you can mass-assign the fields, run the validations and save in just a couple of method calls.
All is well then. Until you need to move from the righteous path. There are some cases when just doing this is not enough, at least if you’re concerned with the health of your project on the long run.
When you need to read some fields that do not belong in the database
Let’s say we have an Event. It will have a start and end time and date. But, to simplify this for the user, we’re just going to require the start date, and the length of the event (in hours), and do the maths ourselves.
We could add a #length_of_the_event temporary field in the Event, and a #before_create callback, like this:
class Event < ActiveRecord::Base
attr_accessor :length_of_the_event
validates :length_of_the_event,
numericality: {greater_than: 0},
on: :create
before_create :calculate_end_dat
def calculate_end_date
self.end_date = self.start_date + length_of_the_event.hours
end
end
Will this work? Of course!
…but the problem is, it is adding logic to the Event class, logic that only makes sense while we are creating the Event using the specific use case from before. The rest of the time we’ll be using the event class for our application, and it will have those methods there, which are of no use to us. Even worst, if you tried to create an Event setting the end date, that would not work. You’d have to add logic to disable the effects of the callback in that case.
Form Object to the rescue
In this case, we could put that logic in a Form Object.
class EventForm < Syrup::FormObject
wraps :event
attribute :length_of_the_event, Integer
validates :length_of_the_event, numericality: {greater_than: 0}
before_validation :before_validation
def before_validation
self.end_date = event.start_date + length_of_the_event.to_i.hours
end
end
Here we moved all the logic for this fields into this FormObject. The Event now has no knowledge of the field, does not need to validate it. Could this logic be added to the controller? Sure. But in that case, you’d have a controller with a lot of logic about how to create the object, on top of handling the parameters and rendering the views.
The resulting form object does not add a lot of code to the overall application, and allow us to separate this logic from the Event class.
External sources
https://github.com/alexsiri7/syrup_form_object/ My implementation of the Form Objects
https://github.com/apotonick/reform Another implementation of Form Objects
https://github.com/brycesenz/freeform Yet another implementation
Rentify Engineering: How important is your job?
11 years
Let’s pause and take a moment to really think about something - your job. How important is your job?
It’s eight hours of your life, five days a week. Let’s think about that for a second. Let’s assume you start working at 16 and you don’t finish until the retirement age of 65. That’s 49…
Pragmatic Thinking and Learning - Refactor Your Wetware
“Alright Brain, you don’t like me, and I don’t like you. But lets just do this, and I can get back to killing you with beer.”
- Homer (Simpson)
Pragmatic Thinking and Learning: Refactor Your Wetware
by Andy Hunt
Magento reindexing, transactions and locks
One of the most frequent errors we got when working with Magento were transactions locking and deadlocking in the database. This usually happened when saving products, and also when customers where making purchases (which prevented them from actually making the purchase, not really something you want!). This locking problems ocurred a lot when we were doing reindexing of the database. ‘SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction’ was an error message we got used to read almost every day.
After some research, we found that the problem in reindexing came primarily from a foreign key in the index_process_event table, which referenced the index_process table. When you did a reindex of any of the indexes, the related row for that index would get locked right away, and stood locked until the transaction was complete (which could, in some cases, take up to 30 minutes, in the case of the url_rewrite index).
Then, when someone was making a purchase, the index_process_event table would get updated, detailing that the product that was being bought, had to be reindex. As that table had a Foreign key to the index_process table, it had to get a read lock (to prevent any changes on the referenced table during the transaction, rendering the result of the transaction inconsistent), which was unable to get until the reindexing was complete.
So, what did we do?
Well, first of all, we got rid of that Foreign Key. Yes, I know, consistency is very important, and an automated check by the database is safe and great. But in this case, it was preventing the business from working, so it was better to remove it. We did some intensive thinking on the matter, and decided that the likelihood of doing an insert that broke the integrity was close to none, and even in the case that it happened, as all reads from that table where done by that very key, the broken record would never be retrieved. This resulted on no more locking because of that index_process table.
Another thing we did, was to change Magento to use Background Reindexing. Out of the box, Magento has 2 reindexing methods: On save, and Manual. On the On save method, every time you update a product or a category, it just goes and updates the related indexes. This provides immediacy, but as it is doing all the changes inside the transaction, and before actually returning a response to the customer, it is very slow to be used. The other method, Manual, just runs a full reindexing of those tables whenever you do it (whether when you clicked on the button on the backend, or by running a command in the console, probably using cron). This is better for when you are not indexing, but when you launch a reindex it pretty much blocks every other activity in the website.
So we implemented Background Reindexing. It works pretty much like On Save reindexing, but instead of doing it inside of the originating transaction, we just stored the task to be done, and then a cron job goes through each of the tasks and reindexes one at a time. It also reindexes each element in it’s own transaction, so the locks do not last for long (usually less than one second). This allows us to parallelize reindexing as well, so the website is usually updated as soon as the changes are made.
UPDATE
This is the code I created to make Background reindexing a reality. Keep in mind that this might cause some problems as the indexing could be done not in order. bg_indexing.tar
I also added a locking mechanism to the cache (I am using Redis now, and were using Memcached, both of them support a way for doing locks) so I could have more than one indexer running simultaneously.
The Indexer::indexEventsCron is called from a cron from inside Magento.
UPDATE 2:
Actually, we started using beanstalkd, which is a task queue, and indexing is being done from there, but the basic principle is the same.
Magento in Multiple servers
Recently, in www.sportpursuit.com, due to the increase on the number of users, we had to migrate from having just one web server that did everything, to having several replicated servers. In order to do that, we run several copies of the website, each on it’s own apache server, all connecting to the same database and memcached server.
This seemed to work well, but we did find some problems:
- When we uploaded images, they were being saved in one of the servers, and the rest did not know about them.
- The load balancer (nginx) was just balancing based on the IPs of the visitors, so it was actually only balancing based on number of users and not on load. If one of the servers started running hot, it would still be getting as many requests as the rest of them.
- The servers were started automatically from an image of the main server, which meant we had to maintain both the main server and the images. And, as they were only started on moments of high load, if there was any problem we would only discover when there was a lot of people accessing the website (effectivelly annoying the greatest amount of people possible).
Therefore, we decided to change a little the setup, creating a dedicated admin server. This way, all admin operations will be done in a centralized server, and will not affect the users experience. It also means that the servers that will run the customers requests, will be identical (less mantainance, and less probabilities of errors on high loads).
How did we do that in Magento?
Well, after some research and debugging, we realized that the admin store in Magento is actually just another store, with a slightly different configuration. So, adding the right configuration to the database, we got to have a different url for the admin server. If you want to do it, all you need to do is add the base_url options in the database to the (scope:websites, scope_id:0) store (which is the admin store). That is of course, in the core_config_data table.
Another thing we had to do, is redirect all requests made to static content (media, skin and js folders) on the website, to the backend (which is the one that has all the images). We achieved that with a rule in nginx, and we also added caching of the resources in nginx (so we don’t need to serve them many times from the backend server).
location ~ ^/(skin|media|js)/{
proxy_pass http://backend_media;
proxy_cache STATIC;
proxy_cache_valid 200 1d;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Protocol https;
}
Finally, we had the problem with the resized images. When you visit a product or category page in Magento, it generates all the images it is going to use on the page. It just creates the resized version of the image, and stores it in the disk for the next request the user is going to make. If it doesn’t find that image, it shows instead a placeholder.
Well, in this distributed servers system, the Magento instances do not have the images there to resize during the rendering of the page. So we had to change the Mage/Catalog/Model/Product/Image.php file, that now, if the image does not exist, it does not try to create the resized version, and still uses the url that would correspond to the resized version of the image.
That will allow Magento to render the right page, all we need nos is a way of generating the resized version of the images. If you check the urls generated, they have this shape:
/media/catalog/product/cache/1/small_image/298x/9df78eab33525d08d6e5fb8d27136e95/t/e/test_image.jpg
/media/catalog/product/cache/1////
So, we just had to create a php file that would create the resized version of the image and store it in the right place, using the image specified with path, and resized to the spec we had. The code for that is:
require_once dirname(__FILE__) . '/app/Mage.php';
Mage::app();
const DEFAULT_CONFIG = "9df78eab33525d08d6e5fb8d27136e95";
$uri = $_SERVER["REQUEST_URI"];
$parts = explode("/",$uri);
$size = (int)$parts[7];
$config = $parts[8];
$path1 = $parts[9];
$path2 = $parts[10];
$path3 = $parts[11];
$file = Mage::getBaseDir("media").DS."catalog". DS. "product"
.DS.$path1.DS.$path2.DS.$path3;
//Reassemble the original path for the image
if ($config == DEFAULT_CONFIG){
//Check that there are no image effects that we should consider
try{
// Load
$processor = new Varien_Image($file);
$processor->keepAspectRatio(1);
$processor->resize($size);
$out_file = Mage::getBaseDir().DS.$uri;
mkdir(dirname($out_file), 0644, true);
$processor->save($out_file);
header("HTTP/1.0 404 Not Found");
} catch(Exception $e){
Mage::logException($e);
header("HTTP/1.0 404 Not Found");
}
} else {
header("HTTP/1.0 404 Not Found");
}
And that is how we managed to have an admin server and several client instances running in parallel.
Testing and Definition of Done
When reading information about Scrum, one of the common themes is the definition of Done. A team is supposed to have a definition of Done before starting a task, as otherwise there is no way of telling when you actually finished the task. That definition of done usually has some particular elements related to that specific task, but there are also some parts of the definition that are common to all of them.
In SportPursuit, our initial definition had the common elements you might find elsewhere: Some degree of testing, and validation from Product Owner. Afterwards, we had to expand it a little:
- Integrated: The task, in order to be completed, has to work with the existing codebase, and cannot break existing functionality.
- Stylized: The resulting user interfaces should match the existing look of the application, and look correct.
- Performant: Ensure the code can be executed in the website without causing performance penalties for visiting users.
- Cache-reviewed: As we use cache heavily in the application, we need to ensure that the resulting code interacts correctly with cache (that is, it does not cache things that should not be cached, and viceversa).
For the last two elements, in order to check them, we had to set up an exact clone of our production server. Testing the performance of an application on a server that is not similar to your production environment is hard (as some things, like memory usage, are more likely to fail in a personal computer than in a live server, when you try to access with 20 people at the same time). And, not having memcache on the PC also creates a small set of differences that makes cache testing more difficult. There are also some other things that are different in the production server (like a CDN, or https, or a backup server that goes live on peak hours) that are close to impossible to check on a local computer. All of those things are easier to check on an integration server.
Problem with Magento Full Page Cache and Redirects

So, in line with the previous post, I want to write about other problem we had when developing the redesign for our website www.sportpursuit.com.
This is a private shopping site, which means you need to signup in order to be able to purchase. So, if someone that was not logged in tries to acces the website, he is redirected to the home page. Then, after logging in, they are redirected to the sales page, where they can see the latest offers we have.
Well, the problem we had was that Magento, trying to cache as much information as possible, was caching the redirect from the sales page to the home page, and also the redirect from the home page to the sales page (effectively causing a redirect loop).
So, what we had to do to fix that problem was:
In the line 501 of app/code/core/Enterprise/PageCache/Model/Processor.php
, where it said
if ($processor && $processor->allowCache($request)) {we changed it to
$headers = $response->getHeaders();
$is_redirect = false;
foreach($headers as $header=>$val){
$is_redirect |= $val["name"]=="Location";
}
if ($processor && $processor->allowCache($request) && !$is_redirect) {
that checks whether the request is a redirect and then prevents the caching of the response.
Two things:
- In order to do it properly, you should either override that file, or create a local copy of it. Do not change Magento’s core!
- This problem is only present in Magento Enterprise (where FullPage cache is). So, if you are using Magento Community, you won’t have this problem! (at least, in MC version 1.7)
Select in Internet Explorer inexplicably closes
In www.sportpursuit.com, before launching the redesign site, we went through a fase of thorough testing. Needless to say, the strangest and hardest to debug errors presented themselves in Internet Explorer.
The strangest one, though, was that some select widgets on the website were closing themselves. That is, you opened them, hover one of the options, and they would just close. This did not happen in all of the website, just on some of them.
Doing some research, all we could find was that http://someguynameddylan.com/lab/ie7-select-hell.php and http://stackoverflow.com/questions/5551288/ies-select-boxes-closes-themselves-when-hovering came across a similar form of the problem.
Well, after a couple of hours of testing, and having disabled all the javascript on the page, I found that the thing that was actually breaking the select was a custom font. We were using that font without a fallback, and that was breaking the Select.
What I believe was happenning, is that when IE tries to render the options, if there is an exception in the styles it might fail and close the options. In this case, it could not render the options, because of the fonts, and in one of the links I sent, was a faulty background declaration.
I hope this helps anyone. And IE rendering developers out there, if you read this, please change the behaviour of the exception catching of the select box to not to do anything, instead of closing the options :).