God tries to help Cain in several ways.
First, He says, "If thou doest well." In other words, He takes away any chance of CXain thinking of it as "Abel versus me." God points to the absolute right and wrong which has existed fromt he foudnation of the world. "Doing well" means to do that which is right.
When God established the proper sacrifice by shedding blood to cover sin, that became the accepted way. Since animals rely on instinct, sacrificing them was different than killing people. People have dominion over animals by virtue of being living, breathing souls. Yes, animals have some instinctive reactions that can be considered to be like feelings. But, they are not nearly as highly developed, and should never be confused with human feelings. The sacrifice is not killed because it has done something wrong. But, this just makes the analogy of Jesus as the Lam of God all the greater He never sinned. And yet, Christ died for the sins of all manking, past, present, and future.
It was right, therefore, for Cain to make a sacrifice like Abel's. It would have been very easy for him to find a sheep and start raising it, or to ask Abel for one. God doesn't expect us to do more than we can with His help. Finding an animal to sacrifice was no huge burden for Cain. Then, he'd have been accepted.
This is the next part of God's plea to Cain. If he does well, he will be accepted. God phrases it so that Cain has to realize that, yes, he will be accepted. People have looked for acceptance since the beginning of time. It's why church doors must always be open, to anyone off the street who wants to be changed within by God's amazing grace. To shut the doors to anyone is a horrible sin. God is holy, but He has also told us to go into all the world and preacht he Gospel to every creature. (Mat. 28:18) He wants everyone to come to Him. The story is told of a man who went to church to steal coats. He was convinced to come in and listen, and he was convicted of his sin. he got saved, and became a wonderful member of the church, serving in numerous areas for many years! It is not we who win souls, it's God.
However, it is never by just being there physically that God accepts us. That man would not have been accepted of God by stealing coats. He was accepted of God by coming to Him with humility, accepting he was a wretched sinner in need of the Savior. God wants Cain to realize the same thing. Yes, he begins with talk of doing well, but remember the context. Doing well, in this case, means the very same thing - recognizing his need for forgiveness, and that God Himself has provided the sacrifice. He called for Cain to come to repentence. he was not asking Cain to do a work, as salvation is never by works, always by faith.
The Hebrew words note the 'excellency" of that acceptance. This goes back to what was said about the prominence of the firstborn male in Hebrew culture. Cain was the firstborn of Adam and Eve's children. The honor of having the line of the Redeemer go through him might have been Cain's if he'd followed God's ways, though we don't know for sure.
Next, God reminds Cain what will happen if he doesn't turn to God for the remission of sin. If he doesn't follow God's way, sin "lieth at the door."
Of course, there had been sins before this - that's why sacrifices were needed. However, this notion of it "lying at the door" means some greater sin (with regard to consequences) than ones he'd committed before this.
Cain had been born with a sin nature. he inherited that from Adam. We all inherit that sin nature. The Adamic nature is what Christ must lovingly guide us away from once we are saved. When we follow that old, Adamic nature, however, we fall into our old, sinful habits. When we keep up in those habits, they become worse and worse, and we lose sight of how Jesus made us new creatures. (2 Cor. 5:17) We are still saved, but it becomes harder and harder for us to hear His Holy Spirit calling us.
That was Cain's problem. The Holy Spiri couldn't indwell people till Christ's blood was shed when He died on Calvary. That's when the veil was torn in two, and the glory of God went from the Holy of Holies in the temple, and spread over all the Earth. (Mat. 27:51) But, it was still possible for God to reach people.
However, Cain faced the very seirous danger of getting so far from God's perfect love, he could have trouble finding his way back.
God explains this with His next statement. He says that "unto thee shall be his desire." Sin, here, is personified. It's given a human attribute - desire - so God can explain the problem more clearly. That thirst for sin - separation from God's perfect love and compassion - would grow, as Cain's desire to sin grew. The possible pain and suffering caused by this sin would grow. It would spiral totally out of control
Finally, God tells Cain what should happen instead. He tells Cain that he should be the one to rule over sin, instead of sin ruling over him.
Charistians are warned of the same thing. The devil walks about like a roaring lion, seeking whom he may devour. (1 Peter 5:8) We are warned many times about the dangers of sin, and that we should put those sinful puarts away. Meanwhile, we can focus on many positive things, such as those in Philippians 4:8.
We can't do it alone, of course. We need Christ's redemption, and we need His Spirit to guide us. Cain couldn't do it on his own, either. He needed God's help, too.
The fact God's appeal to Cain doesn't work is sad. However, it is still the right model to follow. Not everything modeled in Scripture is a good example to follow. Some really bad people have lived. But, God's way is always the one to follow, becuase He is perfect goodness.
Beside, not every outreach will be successful at that moment. But, some will bear fruit years later, that would only have worked with the outreach at the time. As an example, a woman a few years back prayed for her husband to be saved for 44 years! He was saved not just by her prayer, but surely by her attitude all those years.
