Is it possible to make the popup on the thumbs_table.php page and not on the gallery page(item_image.php)?

Yes, that is possible, with some "intelligent" cut & paste   

When you go via your RSGallery2 menu item to a gallery and have the thumbs view set in the table way it is indeed the code in that thumbs_table.php that shows the thumb image(s) with a link to the individual image. This is the code:
<div class="shadow-box" >
<div class="img-shadow">
<a href="<?php echo JRoute::_( "index.php?option=com_rsgallery2&page=inline&id=".$item->id ); ?>">
<img src="<?php echo $thumb->url();?>" alt="<?php echo htmlspecialchars(stripslashes($item->descr), ENT_QUOTES); ?>"/>
</a>
</div>
</div>
You can replace this by, for example
<img src="<?php echo $thumb->url();?>" alt="<?php echo htmlspecialchars(stripslashes($item->descr), ENT_QUOTES); ?>"/>
to check that with this new line you only have the image, without the CSS shadow box and without a link to another page.

You want a popup when you click this image, and ideed, the code from item_image.php is a nice example.

The modal popup behaviour is done with this code, however, $imageOrigingalURL needs to be adjusted from the name that is available in thumb_table.php, together with some other code that is used for that variable. So in the end you can comment/remove the divs for the shadow-box as shown above and put some php code for the modal popup there:
<?php
$watermark = $rsgConfig->get('watermark');
$imageOriginalUrl = $watermark ? waterMarker::showMarkedImage( $item->name, 'original' ) : imgUtils::getImgOriginal( $item->name );
JHTML::_('behavior.modal');
$jsModal = '
window.addEvent("domready", function() {
SqueezeBox.initialize({});
var img = $$("img.rsg2-displayImage")[0];
img.addEvent("click", function(e)
{
new Event(e).stop();
SqueezeBox.fromElement(img,{url:"' . $imageOriginalUrl .'", 
classWindow:"rsg2", 
classOverlay:"rsg2",
onOpen:function(img){
var p = new Element("p", {class:"rsg2-popup-title"});
$(p).appendText("'. strip_tags($item->name) . '");
$(p).inject(img);
var pSize1 = $(p).getSize().size;
p = new Element("p", {class:"rsg2-popup-description"});
$(p).appendText("'. strip_tags($item->descr) . '");
$(p).inject(img);
var pSize2 = $(p).getSize().size;
size = SqueezeBox.size;
size.y += pSize1.y + pSize2.y;
SqueezeBox.resize(size, true);
}});
});
});';
$doc =& JFactory::getDocument();
$doc->addScriptDeclaration($jsModal);
?>
and of course, after that you show the image with with the class (since that is described in the modal code as the img class where the popup should be used):
<img class="rsg2-displayImage" src="<?php echo $thumb->url();?>" alt="<?php echo htmlspecialchars(stripslashes($item->descr), ENT_QUOTES); ?>"/>

Note that when you upgrade RSGallery2 the template Semantice (that holds the files mentioned here) will be overwritten and you'll loose your changes. However, this post will help you set up your own template: http://www.rsgallery2.nl/display_and_templates/how_create_your_own_template_or_modify_semantic_template_16758.0.html.